| 13 Aug 2024 |
 kjeremy | Are there any ideas for reducing evaluation (or flake checks?) memory overhead? I'm consistently OOMing in my flake on a nix flake check. | 16:56:45 |
|  jul1u5 left the room. | 21:55:55 |
| 14 Aug 2024 |
 jade_ | In reply to @kjeremy:matrix.org
Are there any ideas for reducing evaluation (or flake checks?) memory overhead? I'm consistently OOMing in my flake on a nix flake check. you could reduce memory usage or make the gc work better in principle but you have a deeper problem that the thunk structures in nixpkgs essentially are, by their very nature, unfreeable because they will always be accessible as long as pkgs is
your actual solution here is probably to use nix-eval-jobs and eat the fact you restart evaluation a couple times to re-determine what set of thunks are actually hot. that's what Hydra does, effectively. (and no you can't just throw out not-recently-touched thunks and recompute them, evaluation is side effectful in allocation ordering that is both detectable and relied upon by code in nixpkgs, horryingly)
| 02:43:57 |
| jade_ | In reply to @kjeremy:matrix.org
Are there any ideas for reducing evaluation (or flake checks?) memory overhead? I'm consistently OOMing in my flake on a nix flake check. * you could reduce memory usage or make the gc work better in principle but you have a deeper problem that the thunk structures in nixpkgs essentially are, by their very nature, unfreeable because they will always be accessible as long as pkgs is
your actual solution here is probably to use nix-eval-jobs and eat the fact you restart evaluation a couple times to re-determine what set of thunks are actually hot. that's what Hydra does, effectively. (and no you can't just throw out not-recently-touched thunks and recompute them without tossing the entire evaluation, evaluation is side effectful in allocation ordering that is both detectable and relied upon by code in nixpkgs, horryingly)
| 02:44:25 |
|  quapka4 joined the room. | 10:52:05 |
| quapka4 | Hi, I've been asked to report I difference in hash calculation when using nix eval --raw for Nix 2.18.5 and 2.32.2. I am running NixOS 24.05 with Nix 2.18.5. I have tried Nix 2.32.2 using:
nix-shell -p nixVersions.latest -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/05bbf675397d5366259409139039af8077d695ce.tar.gz
``` from Lazamar's website.
For my Flake I observe this difference:
❯ nix --version
nix (Nix) 2.18.5
❯ nix eval --raw
/nix/store/22xdp8sxgn2ph81fl2azrvjzvdv1w7d3-<pkg>-0.3.3
❯ nix eval --raw '.?submodules=1'
/nix/store/r5fjiccnkk3frrlaj0g5w0z1wabiv4n0-<pkg>-0.3.3
❯ nix --version
nix (Nix) 2.23.2
❯ nix eval --raw
/nix/store/swb1a3yz3qhmsml1pjbwlpfx7dsjcbf6-<pkg>-0.3.3
❯ nix eval --raw '.?submodules=0'
/nix/store/swb1a3yz3qhmsml1pjbwlpfx7dsjcbf6-<pkg>-0.3.3
❯ nix eval --raw '.?submodules=1'
/nix/store/swb1a3yz3qhmsml1pjbwlpfx7dsjcbf6-<pkg>-0.3.3
| 10:53:58 |
| quapka4 | * Hi, I've been asked to report I difference in hash calculation when using nix eval --raw for Nix 2.18.5 and 2.32.2. I am running NixOS 24.05 with Nix 2.18.5. I have tried Nix 2.32.2 using:
nix-shell -p nixVersions.latest -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/05bbf675397d5366259409139039af8077d695ce.tar.gz
``` from Lazamar's website.
For my Flake I observe this difference:
❯ nix --version
nix (Nix) 2.18.5
❯ nix eval --raw
/nix/store/22xdp8sxgn2ph81fl2azrvjzvdv1w7d3-<pkg>-0.3.3
❯ nix eval --raw '.?submodules=1'
/nix/store/r5fjiccnkk3frrlaj0g5w0z1wabiv4n0-<pkg>-0.3.3
❯ nix --version
nix (Nix) 2.23.2
❯ nix eval --raw
/nix/store/swb1a3yz3qhmsml1pjbwlpfx7dsjcbf6-<pkg>-0.3.3
❯ nix eval --raw '.?submodules=0'
/nix/store/swb1a3yz3qhmsml1pjbwlpfx7dsjcbf6-<pkg>-0.3.3
❯ nix eval --raw '.?submodules=1'
/nix/store/swb1a3yz3qhmsml1pjbwlpfx7dsjcbf6-<pkg>-0.3.3
| 10:54:15 |
| quapka4 | * Hi, I've been asked to report I difference in hash calculation when using nix eval --raw for Nix 2.18.5 and 2.32.2. I am running NixOS 24.05 with Nix 2.18.5. I have tried Nix 2.32.2 using:
nix-shell -p nixVersions.latest -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/05bbf675397d5366259409139039af8077d695ce.tar.gz
from Lazamar's website.
For my Flake I observe this difference:
❯ nix --version
nix (Nix) 2.18.5
❯ nix eval --raw
/nix/store/22xdp8sxgn2ph81fl2azrvjzvdv1w7d3-\<pkg>-0.3.3
❯ nix eval --raw '.?submodules=1'
/nix/store/r5fjiccnkk3frrlaj0g5w0z1wabiv4n0-\<pkg>-0.3.3
❯ nix --version
nix (Nix) 2.23.2
❯ nix eval --raw
/nix/store/swb1a3yz3qhmsml1pjbwlpfx7dsjcbf6-\<pkg>-0.3.3
❯ nix eval --raw '.?submodules=0'
/nix/store/swb1a3yz3qhmsml1pjbwlpfx7dsjcbf6-\<pkg>-0.3.3
❯ nix eval --raw '.?submodules=1'
/nix/store/swb1a3yz3qhmsml1pjbwlpfx7dsjcbf6-\<pkg>-0.3.3
| 10:54:36 |
| quapka4 | * Hi, I've been asked to report I difference in hash calculation when using nix eval --raw for Nix 2.18.5 and 2.32.2. I am running NixOS 24.05 with Nix 2.18.5. I have tried Nix 2.32.2 using:
nix-shell -p nixVersions.latest -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/05bbf675397d5366259409139039af8077d695ce.tar.gz
from Lazamar's website. For my Flake I observe this difference:
❯ nix --version
nix (Nix) 2.18.5
❯ nix eval --raw
/nix/store/22xdp8sxgn2ph81fl2azrvjzvdv1w7d3-\<pkg>-0.3.3
❯ nix eval --raw '.?submodules=1'
/nix/store/r5fjiccnkk3frrlaj0g5w0z1wabiv4n0-\<pkg>-0.3.3
❯ nix --version
nix (Nix) 2.23.2
❯ nix eval --raw
/nix/store/swb1a3yz3qhmsml1pjbwlpfx7dsjcbf6-\<pkg>-0.3.3
❯ nix eval --raw '.?submodules=0'
/nix/store/swb1a3yz3qhmsml1pjbwlpfx7dsjcbf6-\<pkg>-0.3.3
❯ nix eval --raw '.?submodules=1'
/nix/store/swb1a3yz3qhmsml1pjbwlpfx7dsjcbf6-\<pkg>-0.3.3
| 10:54:52 |
| quapka4 | * Hi, I've been asked to report I difference in hash calculation when using nix eval --raw for Nix 2.18.5 and 2.32.2. I am running NixOS 24.05 with Nix 2.18.5. I have tried Nix 2.32.2 using:
nix-shell -p nixVersions.latest -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/05bbf675397d5366259409139039af8077d695ce.tar.gz
from Lazamar's website. For my Flake I observe this difference:
❯ nix --version
nix (Nix) 2.18.5
❯ nix eval --raw
/nix/store/22xdp8sxgn2ph81fl2azrvjzvdv1w7d3-\<pkg>-0.3.3
❯ nix eval --raw '.?submodules=1'
/nix/store/r5fjiccnkk3frrlaj0g5w0z1wabiv4n0-\<pkg>-0.3.3
❯ nix --version
nix (Nix) 2.23.2
❯ nix eval --raw
/nix/store/swb1a3yz3qhmsml1pjbwlpfx7dsjcbf6-\<pkg>-0.3.3
❯ nix eval --raw '.?submodules=0'
/nix/store/swb1a3yz3qhmsml1pjbwlpfx7dsjcbf6-\<pkg>-0.3.3
❯ nix eval --raw '.?submodules=1'
/nix/store/swb1a3yz3qhmsml1pjbwlpfx7dsjcbf6-\<pkg>-0.3.3
| 10:55:03 |
| quapka4 | I can give you the Flake.{nix, lock} if you wish. | 10:55:18 |
|  Frank Geusch changed their display name from Master Fudge to Frank Geusch. | 13:25:56 |
 Robert Hensing (roberth) | quapka4: we've changed git fetching to bite the bullet and solve its reproducibility problems in 2.19. I gather your repo is private. Could you add #outPath to your flakerefs and compare the store path contents produced by the git fetcher? | 13:42:39 |
| Robert Hensing (roberth) | * quapka4: we've changed git fetching to bite the bullet and solve its reproducibility problems (since 2.19). I gather your repo is private. Could you add #outPath to your flakerefs and compare the store path contents produced by the git fetcher? | 13:43:01 |
| quapka4 | In reply to @roberthensing:matrix.org
quapka4: we've changed git fetching to bite the bullet and solve its reproducibility problems (since 2.19). I gather your repo is private. Could you add #outPath to your flakerefs and compare the store path contents produced by the git fetcher? Nah, it's github.com/quapka/ectester.git, in the build-with-nix branch. | 13:44:04 |
| quapka4 | In reply to @roberthensing:matrix.org
quapka4: we've changed git fetching to bite the bullet and solve its reproducibility problems (since 2.19). I gather your repo is private. Could you add #outPath to your flakerefs and compare the store path contents produced by the git fetcher? * Nah, it's https;//github.com/quapka/ectester.git, in the build-with-nix branch. | 13:44:13 |
| quapka4 | * Nah, it's https://github.com/quapka/ectester.git, in the build-with-nix branch. | 13:44:20 |
| quapka4 | In reply to @roberthensing:matrix.org
quapka4: we've changed git fetching to bite the bullet and solve its reproducibility problems (since 2.19). I gather your repo is private. Could you add #outPath to your flakerefs and compare the store path contents produced by the git fetcher? Sorry, what exactly do you mean with adding #outPath to flakerefs? | 13:49:39 |
| quapka4 | You mean build particular output and check what paths it corresponds to in the store? 🤔 | 13:50:24 |
| kjeremy | In reply to @jade_:matrix.org
you could reduce memory usage or make the gc work better in principle but you have a deeper problem that the thunk structures in nixpkgs essentially are, by their very nature, unfreeable because they will always be accessible as long as pkgs is
your actual solution here is probably to use nix-eval-jobs and eat the fact you restart evaluation a couple times to re-determine what set of thunks are actually hot. that's what Hydra does, effectively. (and no you can't just throw out not-recently-touched thunks and recompute them without tossing the entire evaluation, evaluation is side effectful in allocation ordering that is both detectable and relied upon by code in nixpkgs, horryingly)
Is there a way to evaluate the entire flake with nix-eval-jobs? | 14:36:49 |
 matthewcroughan | Las: I mean previously it was printed once for a nix build but now it is printed over 30 times, for example | 15:27:33 |
 Las | In reply to @matthewcroughan:defenestrate.it
Las: I mean previously it was printed once for a nix build but now it is printed over 30 times, for example I don’t think this is my fault. The code has had many changes recently though, so you could probably check recent commits on that file. | 15:28:18 |
| Robert Hensing (roberth) | In reply to @quapka4:matrix.org
You mean build particular output and check what paths it corresponds to in the store? 🤔 Check whether the change comes from the fetching, and if so figure out what is the difference. If that's what's happened, it's the easiest way to tell the difference. Otherwise you could nix-diff the .drv files to see what changed | 16:30:32 |
| matthewcroughan | Getting a lot more segfaults on Nix 2.24.1:
[427950.394360] nix[2944108]: segfault at b ip 00007fa37f4e7eef sp 00007ffc0d3f3420 error 4 in libgc.so.1.5.3[7fa37f4d1000+1e000] likely on CPU 1 (core 1, socket 0)
[427950.394374] Code: 8d 3d 35 22 01 00 48 8d 0c 49 48 c1 e1 04 48 8b 84 de d8 15 00 00 48 8b 0c 0f 48 8d 0c c1 4c 8b 21 4d 85 e4 0f 84 cb 00 00 00 <49> 8b 3c 24 48 89 39 85 ed 75 26 48 c1 e0 04 48 01 46 40 85 d2 75
[428102.596958] nix[2944942]: segfault at b ip 00007f28e5fbeeef sp 00007ffd714db9a0 error 4 in libgc.so.1.5.3[7f28e5fa8000+1e000] likely on CPU 6 (core 2, socket 0)
[428102.596976] Code: 8d 3d 35 22 01 00 48 8d 0c 49 48 c1 e1 04 48 8b 84 de d8 15 00 00 48 8b 0c 0f 48 8d 0c c1 4c 8b 21 4d 85 e4 0f 84 cb 00 00 00 <49> 8b 3c 24 48 89 39 85 ed 75 26 48 c1 e0 04 48 01 46 40 85 d2 75
[428442.052601] nix[2969540]: segfault at b ip 00007f13977beeef sp 00007ffe40fc2850 error 4 in libgc.so.1.5.3[7f13977a8000+1e000] likely on CPU 1 (core 1, socket 0)
[428442.052613] Code: 8d 3d 35 22 01 00 48 8d 0c 49 48 c1 e1 04 48 8b 84 de d8 15 00 00 48 8b 0c 0f 48 8d 0c c1 4c 8b 21 4d 85 e4 0f 84 cb 00 00 00 <49> 8b 3c 24 48 89 39 85 ed 75 26 48 c1 e0 04 48 01 46 40 85 d2 75
[428868.656235] nix[3006565]: segfault at b ip 00007f80bd39aeef sp 00007ffea6bbdc10 error 4 in libgc.so.1.5.3[7f80bd384000+1e000] likely on CPU 4 (core 0, socket 0)
[428868.656247] Code: 8d 3d 35 22 01 00 48 8d 0c 49 48 c1 e1 04 48 8b 84 de d8 15 00 00 48 8b 0c 0f 48 8d 0c c1 4c 8b 21 4d 85 e4 0f 84 cb 00 00 00 <49> 8b 3c 24 48 89 39 85 ed 75 26 48 c1 e0 04 48 01 46 40 85 d2 75
[429441.941985] nix[3037918]: segfault at b ip 00007f4a2b94ceef sp 00007ffceeddc570 error 4 in libgc.so.1.5.3[7f4a2b936000+1e000] likely on CPU 4 (core 0, socket 0)
[429441.941998] Code: 8d 3d 35 22 01 00 48 8d 0c 49 48 c1 e1 04 48 8b 84 de d8 15 00 00 48 8b 0c 0f 48 8d 0c c1 4c 8b 21 4d 85 e4 0f 84 cb 00 00 00 <49> 8b 3c 24 48 89 39 85 ed 75 26 48 c1 e0 04 48 01 46 40 85 d2 75
[429497.775939] nix[3038329]: segfault at b ip 00007f0675020eef sp 00007fffa0850100 error 4 in libgc.so.1.5.3[7f067500a000+1e000] likely on CPU 4 (core 0, socket 0)
[429497.775951] Code: 8d 3d 35 22 01 00 48 8d 0c 49 48 c1 e1 04 48 8b 84 de d8 15 00 00 48 8b 0c 0f 48 8d 0c c1 4c 8b 21 4d 85 e4 0f 84 cb 00 00 00 <49> 8b 3c 24 48 89 39 85 ed 75 26 48 c1 e0 04 48 01 46 40 85 d2 75
| 18:04:51 |
| matthewcroughan | Happens when nil spawns nix for lsp stuff | 18:05:24 |
| matthewcroughan | Caught nil's perspective: LSP[nil_ls] Failed to evaluate NixOS options: Nix eval failed with signal: 11 (SIGSEGV) (core dumped). Stderr: | 19:41:53 |
 niksnut | Can you get a stacktrace? (E.g. coredumpctl gdb then bt) | 20:34:03 |
| matthewcroughan | https://termbin.com/miur | 21:20:16 |
|  Aurora Ennie Seidr (she / her) joined the room. | 21:25:03 |
| jade_ | In reply to @kjeremy:matrix.org
Is there a way to evaluate the entire flake with nix-eval-jobs? nix-eval-jobs -j4 --flake . | 22:20:48 |
