NixOS CUDA - Public Room Timeline

	NixOS CUDA	281 Members
	CUDA packages maintenance and support in nixpkgs \| https://github.com/orgs/NixOS/projects/27/ \| https://nixos.org/manual/nixpkgs/unstable/#cuda	57 Servers

Load older messages

Sender	Message	Time
11 Nov 2025
Ari Lotter	here we go again :)	21:11:13
Ari Lotter	`nix-shell -p --run` over `nix run --` ? :p	21:12:32
Daniel Fahey	Old habits die hard!	21:13:24
Daniel Fahey	`nix run nixpkgs#dool -- -tcms --swap --top-mem --top-cpu --display` nice	21:19:28
Ari Lotter	lol it's still building jax for whatever reason	21:27:49
Ari Lotter	🫡	21:27:53
Daniel Fahey	`master` must be ahead of `unstable-nixos` (the target I think Flox build against)	21:28:45
Gaétan Lepage	yes.	21:47:41
Gaétan Lepage	We are following `nixos-unstable-small` on `https://cache.nixos.org`	21:48:26
Daniel Fahey	thanks, is that defined by https://github.com/NixOS/infra/blob/809088c05d04849e3660b22fa9e5bc895570c5fe/channels.nix#L32 ?	22:16:35
12 Nov 2025
Ari Lotter	holy moly. 2tb of ram was not enough for JAX. guess i ran too many jobs at once/	03:16:02
Ari Lotter	seems to have perma-stalled at `building python3.12-jax-0.8.0 (pytestCheckPhase): replacing crashed worker gw3`	03:16:19
Ari Lotter	will try again tomorrow :<	03:29:01
Ari Lotter	thanks JAX	03:29:02
Daniel Fahey	That entropy ain't gonna' to increase by itself 😜 2048 GiB / 128 (cores) = 16 GiB per core So JAX must need more than that. So even a rig the price of a house can benefit from swap! I	08:45:31
Daniel Fahey	* That entropy ain't gonna' to increase by itself 😜 2048 GiB / 128 (cores) = 16 GiB per core So JAX must need more than that. So even a rig the price of a house can benefit from swap!	08:45:39
Daniel Fahey	* That entropy ain't gonna' to increase by itself 😜 2048 GiB / 128 cores = 2^11 GiB / 2^7 cores = 2^4 GiB per core = 16 GiB per core So JAX must need more than that. So even a rig the price of a house can benefit from swap!	08:48:52
Gaétan Lepage	If I remember correctly, I can build jax on 64 cores with 128 GB of RAM	09:14:48
Daniel Fahey	* That entropy ain't gonna' increase by itself 😜 2048 GiB / 128 cores = 2^11 GiB / 2^7 cores = 2^4 GiB per core = 16 GiB per core So JAX must need more than that. So even a rig the price of a house can benefit from swap!	11:33:21
Daniel Fahey	128 GiB / 64 cores = 2 GiB per core 🤔 Aha! `replacing crashed worker` appears in my logs shared in https://github.com/NixOS/nixpkgs/issues/445824 Must be a quirk with Intel CPUs then!	12:04:02
Daniel Fahey	* 128 GiB / 64 cores = 2 GiB per core 🤔 Aha! `replacing crashed worker` appears in my logs shared in https://github.com/NixOS/nixpkgs/issues/445824 Must be a quirk with (some, server grade) Intel CPUs then!	12:05:14
Daniel Fahey	* 128 GiB / 64 cores = 2 GiB per core 🤔 Aha! `replacing crashed worker` appears in my logs shared in https://github.com/NixOS/nixpkgs/issues/445824 Must be a quirk with (some, server grade) Intel CPUs then?	12:05:26
Gaétan Lepage	`replacing crashed worker` is `pytest`'s message following one of its processes crashing.	12:12:09
Daniel Fahey	Yeah, could be for any reason, still seems a bit fishy	12:16:40
Gaétan Lepage	Try with `-j 8` for instance. I have never experienced flakiness when building `jax`, even though I have done it dozens of times.	12:17:35
Gaétan Lepage	Never tried on Intel though...	12:17:44
Daniel Fahey	I also only saw build failures with python3.12 Looks like this particular version is in the team cache, so it built fine $ nix-build https://github.com/daniel-fahey/nixpkgs/archive/fix/python3Packages.vllm.tar.gz --pure --arg config '{ allowUnfree = true; cudaSupport = true; }' --attr python312Packages.jax these 3 paths will be fetched (443.62 MiB download, 443.62 MiB unpacked): /nix/store/s2w4h19yylw9ls7q84j8bd1md62kcrzh-python3.12-jax-0.8.0 /nix/store/78kfvx2q26r5053pkp4g9f9y41hc99xm-python3.12-jax-cuda12-pjrt-0.8.0 /nix/store/3a5iw8yqhsc5x16wllsanyyyzqm3xmvd-python3.12-jax-cuda12-plugin-0.8.0 copying path '/nix/store/78kfvx2q26r5053pkp4g9f9y41hc99xm-python3.12-jax-cuda12-pjrt-0.8.0' from 'https://cache.nixos-cuda.org'... copying path '/nix/store/3a5iw8yqhsc5x16wllsanyyyzqm3xmvd-python3.12-jax-cuda12-plugin-0.8.0' from 'https://cache.nixos-cuda.org'... copying path '/nix/store/s2w4h19yylw9ls7q84j8bd1md62kcrzh-python3.12-jax-0.8.0' from 'https://cache.nixos-cuda.org'... /nix/store/s2w4h19yylw9ls7q84j8bd1md62kcrzh-python3.12-jax-0.8.0 Latest build with the matching hash https://hydra.nixos-cuda.org/build/8123#tabs-buildsteps `atlas.nixos-cuda.org` and https://github.com/nixos-cuda/infra/blob/f150adfab4863b131ef67fb97919fff949793995/hosts/atlas/hardware.nix#L28 suggests it's Intel? 🤔	12:32:38
Gaétan Lepage	Then maybe some flakiness with your specific CPU... Sometimes things are weird.	12:37:39
Gaétan Lepage	(`atlas` is configured with `cores = 9`, maybe this plays a role in building `jax` successfully)	12:38:10
Daniel Fahey	Same exact hashed versions declared from `nixos-unstable` and `master` branches too: [daniel@laptop:~]$ nix-build https://github.com/NixOS/nixpkgs/archive/nixos-unstable.tar.gz --pure --arg config '{ allowUnfree = true; cudaSupport = true; }' --attr python312Packages.jax --builders '' unpacking 'https://github.com/NixOS/nixpkgs/archive/nixos-unstable.tar.gz' into the Git cache... these 3 paths will be fetched (443.62 MiB download, 443.62 MiB unpacked): /nix/store/s2w4h19yylw9ls7q84j8bd1md62kcrzh-python3.12-jax-0.8.0 /nix/store/78kfvx2q26r5053pkp4g9f9y41hc99xm-python3.12-jax-cuda12-pjrt-0.8.0 /nix/store/3a5iw8yqhsc5x16wllsanyyyzqm3xmvd-python3.12-jax-cuda12-plugin-0.8.0 copying path '/nix/store/78kfvx2q26r5053pkp4g9f9y41hc99xm-python3.12-jax-cuda12-pjrt-0.8.0' from 'https://cache.nixos-cuda.org'... copying path '/nix/store/3a5iw8yqhsc5x16wllsanyyyzqm3xmvd-python3.12-jax-cuda12-plugin-0.8.0' from 'https://cache.nixos-cuda.org'... copying path '/nix/store/s2w4h19yylw9ls7q84j8bd1md62kcrzh-python3.12-jax-0.8.0' from 'https://cache.nixos-cuda.org'... /nix/store/s2w4h19yylw9ls7q84j8bd1md62kcrzh-python3.12-jax-0.8.0 [daniel@laptop:~]$ nix-build https://github.com/NixOS/nixpkgs/archive/master.tar.gz --pure --arg config '{ allowUnfree = true; cudaSupport = true; }' --attr python312Packages.jax --builders '' unpacking 'https://github.com/NixOS/nixpkgs/archive/master.tar.gz' into the Git cache... /nix/store/s2w4h19yylw9ls7q84j8bd1md62kcrzh-python3.12-jax-0.8.0	12:38:15

Show newer messages

Back to Room ListRoom Version: 9