I cooked up a playground project for shard benchmarking in case that's of interest to anyone.
https://github.com/aakropotkin/shard-test

Right now it just generates a unit/XXXX/* directory structure for every Nixpkgs' derivation. There's just a dummy.nix file in each dir now that I'm going to use test against.

This is likely useful for testing git performance as well.

The differences I'm seeing are basically noise level on Darwin with a pretty naive timer. I wouldn't put too much weight into that but it's a good smell test. With the readFileType patch after averaging a things out I'm ~5% faster ( 16,000+ packages, 100+ runs, no caches ); but the variance it too high to really interpret anything there.

I will put it through perf in a bit

This is starting to make me think that the difference I saw between Darwin and Linux may have just been encryption of the Nix store honestly. Copying the repo to the store takes an extra 30 seconds in the test repo. Reading directories takes under a second. The checksum only takes ~2.5 seconds.

The repo I was griping about had 5x as many files. Which would just about explain why it was taking me 1-5 minutes to do any evaluations.

{ system ? builtins.currentSystem }: let
  nixpkgs = builtins.getFlake "nixpkgs/95aeaf83c247b8f5aa561684317ecd860476fcd6";
  pkgsFor = nixpkgs.legacyPackages.${system};
in pkgsFor."_0verkill"