I was sent here after seeking the wisdom of "the fixed-point/overlay grey-beards".

I'm 3-4 months down a design rabbit hole and I think it's about time to come talk to the folks who have shot themselves in the foot a few times before I injure myself. I'm not a JS user, honestly don't particularly like web stuff; but work asked me to "make Nix+Node tooling that the an intern could handle - you can open source it, ~6 month timeline". I'm close but I know myself and I've been playing with "fun" fixed points for like 2 weeks and if one of you don't save me from myself... 😅

I'll just include my original post.

If you are or know who are the wizards who wrote some of the Haskell newScope or related overlays - I would like to formally request counsel with the elders.
I've been working on some open source Nix+Node.js tooling for ~3 months, and I've accomplished quite a bit; but I'm at this stage where I'm going off the deep end with fixes/extends to basically create crawlers for package metadata. It's a blast; but I need to consult with people who have shot their feet a few times with this stuff because it can turn into a real time sink and I'm going to be out of toes soon. Haskell here is just an example because it is abundantly clear that those authors also went - fully off the deep end; but honestly if you've built any of the fixed-points or overlay systems in nixpkgs let me know.
My puzzle for today has basically been "when should I stack up raw overlays, and at which breakpoints in a pipeline do I compose and apply them". I sort arrived at the realization that a break between "gather pure metadata", optionally "gather impure metadata", "generate stage 1", and "get wild with a scoped callPackage style final round" are sensible; but deciding which of those are exposed for users is and experimenting more is getting nasty because I'm so deep down the path on certain patterns. I need a guide.
I went from "keep it simple stupid" a month ago to "absolutely every piece of data needs to be overridable" once I started collecting all of my functions/builders into an API and a part of me sort of knows "I have loaded a large caliber footgun that is going to go off soon" lol.

Currently the __add and __update functors are more or less how I'm avoiding the complexity of "composing" overlays or fooling with priorities. It just that I can't be lazy and compose 1-4 different options for how to fetch a field into one object and "let the magic happen" which kind of sucks, but I'm not sure that stacking up and composing overlays is without its own set of frustrations.

That's one thing I was hoping to get guidance on honestly before I spend a couple of days chasing down that rabbit hole.

Currently the __add and __update functors are more or less how I'm avoiding the complexity of "composing" overlays or fooling with priorities. It's just that I can't be lazy and compose 1-4 different options for how to fetch a field into one object and "let the magic happen" which kind of sucks, but I'm not sure that stacking up and composing overlays is without its own set of frustrations.

That's one thing I was hoping to get guidance on honestly before I spend a couple of days chasing down that rabbit hole.

In reply to @growpotkin1:matrix.org
Im nesting 4 layers of those scopes for

Nixpkgs
Node Packages
Node Package "outputs"
Package Meta

The two outer layers are effectively a `callPackage` style scope, while the inner two are fixed points that crawl metadata and form builders.

I've been iterating on a variant of this, called using that operates like this:

using pkgs {
          hello-go = ./pkgs/hello-go;
          hello-perl = ./pkgs/hello-perl;
          hello-rust = ./pkgs/hello-rust;
          haskellPackages = {
            hello-haskell-library = ./pkgs/haskellPackages/hello-haskell-library;
          };
          hello-haskell = ./pkgs/hello-haskell;
          python3Packages = using pkgs.python3Packages {
            hello-python-library = ./pkgs/python3Packages/hello-python-library;
          };
          hello-python = ./pkgs/hello-python;

}

where those libraries are available in the packages that need them and everything is automatically callPackage'd correctly. The end result is then only the packages referenced, not all of Nixpkgs, even though it can access it. It's like a recursiveCallPackageWith. The regular structure here also makes it easy to automatically create that entire call automatically by reading directories/files.

In reply to @growpotkin1:matrix.org
Im nesting 4 layers of those scopes for

Nixpkgs
Node Packages
Node Package "outputs"
Package Meta

The two outer layers are effectively a `callPackage` style scope, while the inner two are fixed points that crawl metadata and form builders.

I've been iterating on a variant of this, called using that operates like this:

using pkgs {
          hello-go = ./pkgs/hello-go;
          hello-perl = ./pkgs/hello-perl;
          hello-rust = ./pkgs/hello-rust;
          haskellPackages = {
            hello-haskell-library = ./pkgs/haskellPackages/hello-haskell-library;
          };
          hello-haskell = ./pkgs/hello-haskell;
          python3Packages = {
            hello-python-library = ./pkgs/python3Packages/hello-python-library;
          };
          hello-python = ./pkgs/hello-python;

}

where those libraries are available in the packages that need them and everything is automatically callPackage'd correctly. The end result is then only the packages referenced, not all of Nixpkgs, even though it can access it. It's like a recursiveCallPackageWith. The regular structure here also makes it easy to automatically create that entire call automatically by reading directories/files.