Currently the __add and __update functors are more or less how I'm avoiding the complexity of "composing" overlays or fooling with priorities. It just that I can't be lazy and compose 1-4 different options for how to fetch a field into one object and "let the magic happen" which kind of sucks, but I'm not sure that stacking up and composing overlays is without its own set of frustrations.

That's one thing I was hoping to get guidance on honestly before I spend a couple of days chasing down that rabbit hole.

Currently the __add and __update functors are more or less how I'm avoiding the complexity of "composing" overlays or fooling with priorities. It's just that I can't be lazy and compose 1-4 different options for how to fetch a field into one object and "let the magic happen" which kind of sucks, but I'm not sure that stacking up and composing overlays is without its own set of frustrations.

That's one thing I was hoping to get guidance on honestly before I spend a couple of days chasing down that rabbit hole.

In reply to @growpotkin1:matrix.org
Im nesting 4 layers of those scopes for

Nixpkgs
Node Packages
Node Package "outputs"
Package Meta

The two outer layers are effectively a `callPackage` style scope, while the inner two are fixed points that crawl metadata and form builders.

I've been iterating on a variant of this, called using that operates like this:

using pkgs {
          hello-go = ./pkgs/hello-go;
          hello-perl = ./pkgs/hello-perl;
          hello-rust = ./pkgs/hello-rust;
          haskellPackages = {
            hello-haskell-library = ./pkgs/haskellPackages/hello-haskell-library;
          };
          hello-haskell = ./pkgs/hello-haskell;
          python3Packages = using pkgs.python3Packages {
            hello-python-library = ./pkgs/python3Packages/hello-python-library;
          };
          hello-python = ./pkgs/hello-python;

}

where those libraries are available in the packages that need them and everything is automatically callPackage'd correctly. The end result is then only the packages referenced, not all of Nixpkgs, even though it can access it. It's like a recursiveCallPackageWith. The regular structure here also makes it easy to automatically create that entire call automatically by reading directories/files.

In reply to @growpotkin1:matrix.org
Im nesting 4 layers of those scopes for

Nixpkgs
Node Packages
Node Package "outputs"
Package Meta

The two outer layers are effectively a `callPackage` style scope, while the inner two are fixed points that crawl metadata and form builders.

I've been iterating on a variant of this, called using that operates like this:

using pkgs {
          hello-go = ./pkgs/hello-go;
          hello-perl = ./pkgs/hello-perl;
          hello-rust = ./pkgs/hello-rust;
          haskellPackages = {
            hello-haskell-library = ./pkgs/haskellPackages/hello-haskell-library;
          };
          hello-haskell = ./pkgs/hello-haskell;
          python3Packages = {
            hello-python-library = ./pkgs/python3Packages/hello-python-library;
          };
          hello-python = ./pkgs/hello-python;

}

where those libraries are available in the packages that need them and everything is automatically callPackage'd correctly. The end result is then only the packages referenced, not all of Nixpkgs, even though it can access it. It's like a recursiveCallPackageWith. The regular structure here also makes it easy to automatically create that entire call automatically by reading directories/files.

Yeah keep in touch. And of course, feel free to expropriate patterns from my libs 🙂

I think exploring this sort of thing may be good for test driving routines that can work their way into Nixpkgs after a bit of battle testing

Alright. My Frankenstein's monster is alive working with my over-engineered overlay system.

I won't pretend that it's "a good model" as is; but it's able to back propagate changes to higher scopes, sort of like a module system does.

Its probably misguided, but 🤷

https://github.com/aameen-tulip/at-node-nix/blob/nps-scoped/pkgs/node-pkg-set.nix

You can feed it whatever node.js garbage you have laying around, and it should give you the same derivation without regard for what "method of input" you gave.

This means if I use pkg A-1.0.0 in one lock or I pull it from a registry or I have a local copy or whatever - cache hits work.
This is the biggest issue with node2nix now - it can't "really" compose node package sets which is why Nixpkgs has to maintain a single instance and they don't recommend packaging modules as separate from CLI tools.

Oh, and I don't call NPM ever and Python only gets used when it's actually needed. Most derivations don't depend on Node even so you won't hog hydra.