I've got a file temp.nix with this expression:

let
  numClosures = 1;
  nixpkgsPath =
    (builtins.getFlake "github:NixOS/nixpkgs/42a63d6a63c0ae7d715e46450fe5a5f69394525d").outPath;
in
builtins.genList (
  _:
  let
    drvPath' = (import (nixpkgsPath + "/nixos/release.nix") { }).closures.gnome.x86_64-linux.drvPath;
  in
  builtins.deepSeq drvPath' drvPath'
) numClosures

I ran nix eval by way of

/usr/bin/env time -v env GC_INITIAL_HEAP_SIZE=512M GC_MAXIMUM_HEAP_SIZE=512M GC_PRINT_STATS=1 NIX_SHOW_STATS=1 nix eval --json --expr "$(cat temp.nix)"

and kept lowering the maximum GC size until it would OOM with a numClosures = 1 (that's how I arrived at 512M). My expectation is that 512M should be sufficient to evaluate many copies of that closure, given I'm taking only the drvPath, and the only increase in memory usage I'd expect between the first and second evaluation of the closure is that the evaluator has allocated memory for the list of derivations it's going to print as JSON. However, when I use numClosures = 2, I OOM. Keeping everything same, I bumped it up to 1G of memory, and I still OOM with numClosures = 2.

One thing which could be tested is to create a large expression in the same fashion, without using import, to see if that has a different memory profile.

I would also like such tooling, but I really don't like reading or writing C/C++ :(

I've only ever read/written C/C++ in anger and I very much consider myself incompetent with C/C++

I haven't looked at or touched the Immer stuff in a long while; the biggest road blocks I saw to moving forward with it were:

1. the issue where the GC would still collect values while they were needed
2. creating wrappers around lists/bindings to abstract over the implementation (and the changes that would cause to the C API), although I think you did or started this with your Value refactoring
3. rewriting relevant portions of the evaluator to use the different implementations offered by Immer (using the mutable APIs where appropriate)
4. making changes to the stats reported by the Nix evaluator
5. the lack of any sort of benchmark suite or platform on which to benchmark making it hard to discern what exactly changed and how

I think 1. was being looked at by the author of Immer, not sure whether they ever got the chance to get into it; 2. and 5. should happen regardless as a code/project hygiene thing.

Fundamentally I still think it's important we look at the data structures Nix offers, the ways in which those data structures are most commonly used, and implementations that make those operations fast :l

In reply to @connorbaker:matrix.org
There’s an import cache

i'm surprised i am missing it when trying to use Nix as a library

ix/store/dxar61b2ig87cfdvsylfcnyz6ajls91v-nix-2.28.3-dev/include/nix/util/fmt.hh:4:10: fatal error: boost/format.hpp: No such file or directory
    4 | #include <boost/format.hpp>
      |          ^~~~~~~~~~~~~~~~~~