In reply to @infinisil:matrix.org
Btw I'm gonna chill in https://meet.jit.si/nixpkgs-architecture, working on https://github.com/nixpkgs-architecture/simple-package-paths/pull/20 a bit based on what we discussed today, if anybody wants to join and help out

I did the million directories test and it ran in exactly the same amount of time with and without the patch.

I think my filesystem on OSX must support some equivalent of dirents which would explain why there's no change.

In any case I'm pretty sure my performance issues were more about encryption of the Nix store.

Still, I think being able to check individual files' types is useful ; but I'd say from a practical standpoint this isn't relevant to the "simple package paths" RFC

so:

let
  inherit (builtins.getFlake "nixpkgs") lib;
  pkgNames = ...;
  pkgs = builtins.listToAttrs ( map ( n: {
    name = n;
    value = lib.callPackagesWith pkgs ( ./unit + "/<SHARD-DIR>/${n}" );
  } ) pkgNames );
in pkgs.hello

Doesn't necessarily cause anything other than the closure of hello to be read IF we are careful in ensuring that those attribute names are explicitly formed from builtins.readDir calls as opposed to something like:

getName = ( import ./unit/<SHARD-DIR>/<PKG-DIR>/pkg-fun.nix { ... } ).meta.name;

so:

let
  inherit (builtins.getFlake "nixpkgs") lib;
  pkgNames = ...;  # lots of `readDir` crawling like I have in "shard-test" repo
  pkgs = builtins.listToAttrs ( map ( n: {
    name = n;
    value = lib.callPackagesWith pkgs ( ./unit + "/<SHARD-DIR>/${n}" );
  } ) pkgNames );
in pkgs.hello

Doesn't necessarily cause anything other than the closure of hello to be read IF we are careful in ensuring that those attribute names are explicitly formed from builtins.readDir calls as opposed to something like:

getName = ( import ./unit/<SHARD-DIR>/<PKG-DIR>/pkg-fun.nix { ... } ).meta.name;

so:

let
  inherit (builtins.getFlake "nixpkgs") lib;
  pkgNames = ...;  # lots of `readDir` crawling like I have in "shard-test" repo
  pkgs = builtins.listToAttrs ( map ( n: {
    name = n;
    value = lib.callPackagesWith pkgs ( ./unit + "/<SHARD-DIR>/${n}" );
  } ) pkgNames );
in pkgs.hello

Doesn't necessarily cause anything other than the closure of hello to be read IF we are careful in ensuring that those attribute names are explicitly formed from builtins.readDir calls as opposed to something like:

# BAD: don't do this
getName = ( import ./unit/<SHARD-DIR>/<PKG-DIR>/pkg-fun.nix { ... } ).meta.name;

for example these names throw a wrench in the approach I'm recommending:

pkgs = {
  foo = ...;
  Foo = ...;
  fOo = ...;
  FOO = ...;
};

Because on case insensitive filesystems you can't have 4 directories with those names. Instead you'd have to either make some encoding for dirnames or write the package name to some file ( which would be slow as sin ).

In the shard-test:gen.nix I had to clobber conflicting names by converting them to lowercase:

{ nixpkgs ? builtins.getFlake "nixpkgs"
, system  ? builtins.currentSystem
, lib     ? nixpkgs.lib
, pkgsFor ? nixpkgs.legacyPackages.${system}
}: let
  keep   = _: v: ( builtins.tryEval ( lib.isDerivation v ) ).success;
  drvs   = lib.filterAttrs keep pkgsFor;
  names  = builtins.attrNames drvs;
  sdirN  = c: n: builtins.substring 0 c ( lib.toLower n );  # <<< Clobbers
  shards = c: builtins.groupBy ( sdirN c ) names;
  defPkg = name: {
    name = lib.toLower name;
    value."default.nix" = ''
      { system ? builtins.currentSystem }: let
        nixpkgs = builtins.getFlake "nixpkgs/${nixpkgs.rev}";
        pkgsFor = nixpkgs.legacyPackages.''${system};
      in pkgsFor."${name}"
    '';
  };
  mkShardDir = ns: builtins.listToAttrs ( map defPkg ns );
  self = {
    unit4 = builtins.mapAttrs ( _: mkShardDir ) ( shards 4 );
    ...
    unit  = self.unit4;
  };
in self