| 12 Nov 2025 |
 Ari Lotter | allllllright let's try nixpkgs-review again with the new binary cache :p | 15:30:38 |
| Ari Lotter | still building jax, but, uhhhh, we ball | 15:40:02 |
 Gaétan Lepage | Which PR? | 15:56:37 |
| Ari Lotter | this one https://github.com/NixOS/nixpkgs/pull/460701 | 15:57:23 |
| Ari Lotter | yep, workers keep crashing :/
] building python3.12-jax-0.8.0 (pytestCheckPhase): replacing crashed worker gw1 | 15:57:38 |
| Ari Lotter | hm,
warning: ignoring the client-specified setting 'sandbox', because it is a restricted setting and you are not a trusted user
warning: ignoring the client-specified setting 'system', because it is a restricted setting and you are not a trusted user
would setting myself as a trusted user fix this, i wonder | 16:02:06 |
 Daniel Fahey | Yeah you might want to use extra-substituters (I never grok'd the difference, if I'm being honest) | 16:06:51 |
| Daniel Fahey | not sure if it's different with Nix on Ubuntu, but on NixOS, I have to rebuild the system before the binary cache is available. Is there a rebuild step with plain Nix? | 16:08:28 |
| Daniel Fahey | I reckon they might be in a private repo hinted at in https://github.com/flox/nixpkgs/pull/3#issuecomment-1276439899
But the https://github.com/flox/nixpkgs/tree/unstable is a simple fork that I'd like to see sync'd/rebased from upstream Nixpkgs more frequently.
Gaétan Lepage is this is the kind of thing that could be discussed in your CUDA Team meetings, and maybe brought up to the Steering Committee for discussion with Flox?
| 16:12:16 |
| Gaétan Lepage | Flox is managing their cache internally. As you pointed, they use an internal fork of nixpkgs that is slightly delayed from nixos-unstable (or nixos-unstable-small). It's normal that their cache is less fresh than chache.nixos-cuda.org. | 16:13:41 |
| Gaétan Lepage | The difference is that they have the permission from Nvidia to redistribute their binaries. | 16:14:01 |
| Daniel Fahey | 😅 | 16:14:45 |
| Ari Lotter | i have it in both because i don't understand it <3 | 16:15:52 |
| Ari Lotter | also - 100% sure i'm not running out of ram anymore, at only ~400gb/2tb used on the machine, but jax still has crashed workers - and not sure if it's progressing | 16:17:00 |
| Ari Lotter | can i pull up interactive logs for its derivation somehow? | 16:17:11 |
| Ari Lotter | oh nice, i got a real failure | 16:17:50 |
| Daniel Fahey | 
Download image.png | 16:18:16 |
| Daniel Fahey | Another option Ari, is you follow the "Help" button next to the closure link at https://hydra.nixos-cuda.org/build/8123
| 16:18:17 |
| Daniel Fahey | Could just pipe output through tee, but normally at the tail of the the crash, it has the command to run to see the log | 16:19:19 |
| Daniel Fahey | I'd be interested to see them, see if they're similar to what I saw on nixbuild.net -> you said it crashed during the test stage? | 16:19:53 |
| Daniel Fahey | *phase | 16:19:56 |
| Daniel Fahey | I also wonder if nixpkgs-review is complicating things | 16:20:33 |
| Ari Lotter | grabbing logs! | 16:22:05 |
| Ari Lotter | Download test.txt | 16:27:01 |
| Ari Lotter | that's a big log lol | 16:27:04 |
| Ari Lotter | but yeah, i hit both some INTERNAL: Failed to materialize symbols and some LLVM compilation error: Cannot allocate memory 🤔 | 16:28:19 |
| Ari Lotter | (keeping in mind i had >1600gb of ram available at all times) | 16:28:52 |
| Daniel Fahey | Might be different than what I saw on nixbuild.net but not sure, did your new attempt crashed building the python3.13 version, so not 3.12 like yesterday
The (server grade) Intel hypothesis still isn't disproven, tbf | 16:40:31 |
| Ari Lotter | 3.12 version is stuck on building python3.12-jax-0.8.0 (pytestCheckPhase): replacing crashed worker gw1 | 16:41:02 |
| Daniel Fahey | Have you fed it into a clanker yet? | 16:41:04 |
