In reply to @lily:lily.flowers

Apologies for possibly incoherent, run-on sentence brain dump (also Winter may also have some thoughts/ideas/plans, but this is my current understanding):

Initially, we need to go ahead and get https://github.com/NixOS/nixpkgs/pull/206476 merged, and then get https://github.com/NixOS/nixpkgs/pull/206477 and https://github.com/NixOS/nixpkgs/pull/214454 cleaned up for merge (and also probably pull in https://github.com/winterqt/nixpkgs/commit/f78408065637e194fce1bfc3820954baef9efb74 to the latter PR before rebasing and cleaning it up)

#206477 needs some minor bitrot figured out and I want to actually migrate the npm patching to the nodejs derivation itself, so handling it between versions is not a hassle and so that we aren't having to juggle getting a patched nodejs in all the right places (like manually prepending it to nativeBuildInputs...). I sorta explained what I was thinking about that here: https://github.com/NixOS/nixpkgs/pull/206477#discussion_r1157922584

The head commit on #214454 and the other commit I linked is needed to fix some problems with assumptions the fetcher makes, both by correcting some assumptions and doing some changes in a lockfile fixup step to map onto some assumptions where reasonable (e.g. why bother fetching a dep more than once just because one node in the dep tree has a different hash type...)

For npm workspaces, dotlambda tried adding an arg to buildNpmPackage which adds a workspace package to build and that actually kinda seems to sorta work (but other monorepo types like lerna and such are probably far-future goals for now because those will basically need their own specialized handling in both fetcher and builder and aren't really that worth it if they are published to registry, since our tooling can already grok most stuff uploaded to the registry, which lerna and such generation sane npm tarballs for that don't require lerna iirc)

We really should add a fetchNpmRegistry for getting a tarball from the registry, kinda like fetchPypi. It's certainly the simplest thing that I would like done, but would be a helpful one for stubborn from-source packages that have acceptable tarballs in the registry

For yarn berry lockfile handling, we need to look closely at the new format and see what's different and if we even need to make that many updates to our existing fetcher. It's possible all it needs to work is to pull the newer lockfile parser (right now it pulls the legacy yarnpkg lockfile parser). We will also need yarn berry packaged: https://github.com/NixOS/nixpkgs/pull/214699

For pnpm lockfile handling, I have not looked at this at all. Ideally it's just a new format to parse and can map onto how we generally make fetchers already (even better: making it another frontend to the existing prefetch-npm-deps fetcher) and then everything after Just Works when called with pnpm (but my experience with nodejs has been every little thing requires a significant amount of debugging hell to untangle...)

We currently only look for package-lock.json in buildNpmPackage for git deps with lifecycle scripts, and I'm not entirely certain yet how we should appropriately handle git deps with heterogeneous lockfiles (we'd probably have to make pacote intelligently call the correct builder I guess depending on what lockfile is available (we're patching npm/pacote already anyway) and maybe make it so prefetch-npm-deps can parse several different lockfiles and fetch extras from git deps properly with different lockfiles). Also some projects lack lockfiles entirely and there is little hope for those. We may want some way to override lockfiles for deps deep in the tree for those and for an escape hatch to support git deps with unsupported lockfiles

We probably will want to continue adding more documentation, to describe corner cases and how to account for them (the diagnostic messages on build failure already do some of that and it's not so bad right now. Some of the above future-stuff will just need it)

In reply to @ambroisie:belanyi.fr
Would love to see pnpm handled soon!

In reply to @hexa:lossy.network
be the change you want to see in the world ig 😄

In reply to @ambroisie:belanyi.fr
Given that that tool exists though, I do believe it should be possible to add it to nixpkgs (somewhat) easily

In reply to @ambroisie:belanyi.fr
NPM has a CA cache?

In reply to @hexa:lossy.network
Lily Foster: with the latest buildNpmPackage changes merged I think I get an endless loop in installPhase 😄