I'm thinking something like this. (For a flat solution)

all-packages = builtin.proxyAttrs {
  lookup = name: pkgs.callPackage (import "pkgs/${name}.nix");
  list-keys = # List all packages and return a list or something here.
};

Then for the common case of nix-build -A foo it just calls lookup "foo" and only that package is loaded.

Slightly related: I just noticed /nix/var/log/nix/drvs/ has a 2 letter prefix sub dir pattern

λ ls /nix/var/log/nix/drvs/
 00   0s   1l   2f   38   42   4w   5n   6h   7a # ... etc ...

In reply to @j-k:matrix.org

Slightly related: I just noticed /nix/var/log/nix/drvs/ has a 2 letter prefix sub dir pattern

λ ls /nix/var/log/nix/drvs/
 00   0s   1l   2f   38   42   4w   5n   6h   7a # ... etc ...

/nix/store is stored locally, rarely listed and mostly accessed by the entry name. That's why we get get away with FS choosing optimal representation for it (hash tree usually).
Nixpkgs is stored in Git and often listed in editors and UIs, all of them are not great with huge directories, so we should optimize for smaller ones.

I wonder why such nested structure was chosen for logs though.

In reply to @kevincox:matrix.org

I'm thinking something like this. (For a flat solution)

all-packages = builtin.proxyAttrs {
  lookup = name: pkgs.callPackage (import "pkgs/${name}.nix");
  list-keys = # List all packages and return a list or something here.
};

Then for the common case of nix-build -A foo it just calls lookup "foo" and only that package is loaded.

I really like that idea. Especially as it allows to add a package by just dropping a file somewhere. Having to edit some all-packages.nix is really unnecessary overhead.

Assuming overrides are still a thing at the time when this gets implemented, we'd have to find a new solution for the default callPackage overrides that are currently done in all-packages.nix for some packages.
I see the following options for this:

1. Create one central overrides file for default callPackage overrides (I think this is a bad idea)
2. Manage the overrides within the .nix of each package, though then it requires a reference to pkgs.
3. Deprecate in-line overrides all-together and enforce the creation of an actual individual package for each required package variation. This would make every existing package variation referenceable by an actual attribute name, which is great for further overriding by the user.

For example, if we have packages hello and foobar, where hello depends on an overridden foobar, we could have the following files:

/fo/ob/foobar.nix
/he/ll/hello.nix
/he/ll/foobar.nix  -  (the overridden foobar package for hello)

/he/ll/foobar.nix could look like this:

{foobar,}:
foobar.overrideAttrs (old: {
   pname = "foobar-for-hello";
})