!NhAsaYbbgmzHtXTPQJ:funklause.de

Nix NodeJS

203 Members
58 Servers

Load older messages

SenderMessageTime
18 Nov 2023
@adis:blad.is@adis:blad.ishttps://github.com/adisbladis/buildNodeModules/blob/master/hooks/node-modules-hook.js#L4903:51:29
@adis:blad.is@adis:blad.is So now I can use npm as a locker, and Nix builds everything in node_modules <3 03:51:52
@adis:blad.is@adis:blad.isAnd the shell hook takes care of keeping symlinks up to date03:52:08
@adis:blad.is@adis:blad.isCreated a readme at https://github.com/adisbladis/buildNodeModules04:16:34
@adis:blad.is@adis:blad.is

I converted github-copilot-cli in nixpkgs and here are the eval performance results:

  • before.json
{
  "cpuTime": 0.7983409762382507,
  "envs": {
    "bytes": 16531624,
    "elements": 828921,
    "number": 618766
  },
  "gc": {
    "heapSize": 402915328,
    "totalBytes": 180765008
  },
  "list": {
    "bytes": 3000672,
    "concats": 21613,
    "elements": 375084
  },
  "nrAvoided": 864119,
  "nrFunctionCalls": 558479,
  "nrLookups": 201175,
  "nrOpUpdateValuesCopied": 4886502,
  "nrOpUpdates": 45948,
  "nrPrimOpCalls": 391174,
  "nrThunks": 1152515,
  "sets": {
    "bytes": 98766016,
    "elements": 5996044,
    "number": 176832
  },
  "sizes": {
    "Attr": 16,
    "Bindings": 16,
    "Env": 16,
    "Value": 24
  },
  "symbols": {
    "bytes": 386981,
    "number": 37425
  },
  "values": {
    "bytes": 36922296,
    "number": 1538429
  }
}
  • after.json
{
  "cpuTime": 0.510466992855072,
  "envs": {
    "bytes": 9614136,
    "elements": 488211,
    "number": 356778
  },
  "gc": {
    "heapSize": 402915328,
    "totalBytes": 91434160
  },
  "list": {
    "bytes": 1416640,
    "concats": 14601,
    "elements": 177080
  },
  "nrAvoided": 511866,
  "nrFunctionCalls": 321155,
  "nrLookups": 126367,
  "nrOpUpdateValuesCopied": 2282699,
  "nrOpUpdates": 28766,
  "nrPrimOpCalls": 240324,
  "nrThunks": 639741,
  "sets": {
    "bytes": 45028896,
    "elements": 2733143,
    "number": 81163
  },
  "sizes": {
    "Attr": 16,
    "Bindings": 16,
    "Env": 16,
    "Value": 24
  },
  "symbols": {
    "bytes": 370066,
    "number": 34762
  },
  "values": {
    "bytes": 20532984,
    "number": 855541
  }
}
10:01:22
@c0ba1t:matrix.orgCobaltAre the results in any way surprising? The cputime looks a bit too small to be used for reliable measurements without perf counters10:41:33
@adis:blad.is@adis:blad.isNot surprising to me. I was expecting it to be a bit faster and use less memory I'm not overly concerned with exact wall time, I'm OK concluding it's a bit faster and leave it at that.11:11:13
@adis:blad.is@adis:blad.isA concern that often comes up when when talking about lock file parsing in nixpkgs performance. I wanted to have ballpark numbers to refer to. 11:12:52
@c0ba1t:matrix.orgCobaltAh, okay. Thank you for sharing11:13:31
@raitobezarius:matrix.orgraitobezarius
In reply to @adis:blad.is
A concern that often comes up when when talking about lock file parsing in nixpkgs performance.
I wanted to have ballpark numbers to refer to.
You know that I nerdsnipped @tomberek on adding the parsing built-in based on a EBNF grammar (a packrat parser) 		11:48:39
@raitobezarius:matrix.orgraitobezariusHe got OK speed I think11:48:46
@raitobezarius:matrix.orgraitobezariusBut he uncovered a quadratic issue with Nix strings11:48:54
@raitobezarius:matrix.orgraitobezariusBecause substr is linear all the time as strings in Nix doesn't really know their own length11:49:14
@adis:blad.is@adis:blad.isHuh11:49:28
@adis:blad.is@adis:blad.is
In reply to @raitobezarius:matrix.org
You know that I nerdsnipped @tomberek on adding the parsing built-in based on a EBNF grammar (a packrat parser)
That would be cool to have. 		11:50:50
@adis:blad.is@adis:blad.isBut for this case you really don't need fancy parsing, you just need json11:51:00
@raitobezarius:matrix.orgraitobezariusBut gigabyte/s JSON parsing:>12:00:28
@raitobezarius:matrix.orgraitobezariusIdk how fast is our JSON library12:00:43
@adis:blad.is@adis:blad.isFatet than our nix parser :p12:06:05
@adis:blad.is@adis:blad.is* Faster than our nix parser :p12:08:01
@adis:blad.is@adis:blad.is

After some further refactoring:

{
  "cpuTime": 0.34559598565101624,
  "envs": {
    "bytes": 6537216,
    "elements": 330112,
    "number": 243520
  },
  "gc": {
    "heapSize": 402915328,
    "totalBytes": 63767152
  },
  "list": {
    "bytes": 985672,
    "concats": 9249,
    "elements": 123209
  },
  "nrAvoided": 358263,
  "nrFunctionCalls": 218872,
  "nrLookups": 82272,
  "nrOpUpdateValuesCopied": 1712032,
  "nrOpUpdates": 18283,
  "nrPrimOpCalls": 171800,
  "nrThunks": 461439,
  "sets": {
    "bytes": 33342256,
    "elements": 2027794,
    "number": 56097
  },
  "sizes": {
    "Attr": 16,
    "Bindings": 16,
    "Env": 16,
    "Value": 24
  },
  "symbols": {
    "bytes": 246484,
    "number": 24916
  },
  "values": {
    "bytes": 13269240,
    "number": 552885
  }
}
13:24:25
@adis:blad.is@adis:blad.isI mainly did it for improved UX, but that was a sweet bonus.13:24:56
@adis:blad.is@adis:blad.is It looks much more similar to fetchNpmDeps in usage now 13:29:54
19 Nov 2023
@zxgu:matrix.orgZXGU joined the room.11:02:39
20 Nov 2023
@whentze:matrix.orgWanja Hentze
In reply to @adis:blad.is
Cool :)
Hah, it doesn't work (at least not out of the box) but not for the reasons I was thinking of 		16:08:31
@whentze:matrix.orgWanja Hentzewe have a .npmrc to do bearer auth against a private gitlab registry and we hacked support for that into our npmlock2nix fork16:09:10
@whentze:matrix.orgWanja Hentze after porting that over (in an even hackier manner), the next issue seems to be that we have packageRoot != src. That makes the npm-config-hook.sh unhappy. Looks like it's being run in src, not in packageRoot for some reason. 17:04:37
@whentze:matrix.orgWanja Hentze debugging a little bit, I found that $npmRoot is not set at this point: https://github.com/adisbladis/buildNodeModules/blob/master/hooks/npm-config-hook.sh#L6 17:09:42
@whentze:matrix.orgWanja Hentze am I still expected to pass sourceRoot to mkDerivation for this to work? 17:17:07
@whentze:matrix.orgWanja Hentze

so atm I have 2 questions adisbladis

  • if I cleaned up the code to parse .npmrc and pass auth headers based on it into fetchurl, would you be willing to take it?
  • is src != packageRoot supposed to work, or is it an unsupported configuration?
17:30:49

Show newer messages

Back to Room ListRoom Version: 6