NixOS CUDA - Public Room Timeline

	NixOS CUDA	288 Members
	CUDA packages maintenance and support in nixpkgs \| https://github.com/orgs/NixOS/projects/27/ \| https://nixos.org/manual/nixpkgs/unstable/#cuda	59 Servers

Load older messages

Sender	Message	Time
31 Oct 2025
Gaétan Lepage	We have very recently acquired new hardware. That is still far from the perfect infra, but it's definitely a good progress.	20:02:54
Gaétan Lepage	I broke the record yesterday building `python3Packages.torch` with `cudaSupport` enabled. -> 41 min on 96 cores.	20:03:48
Gaétan Lepage	Do not try to replicate on your laptop 🫠	20:04:03
Gaétan Lepage	connor (burnt/out) (UTC-7) ACK for both.	20:04:17
Robbie Buxton	I’ve oomed a machine with over 1tb of ram building nix cuda packages 😎	20:04:38
apyh	In reply to @glepage:matrix.org I broke the record yesterday building `python3Packages.torch` with `cudaSupport` enabled. -> 41 min on 96 cores. omg. i wanna try.	20:04:40
Gaétan Lepage	I have only 128GB of RAM on my builder. So I got swap to a (sometimes necessary) 500GB size.	20:05:45
Gaétan Lepage	`ptxas` can be very expensive memory-wise...	20:06:15
Gaétan Lepage	connor (burnt/out) (UTC-7) I found out the issue with `firefox`. Both prior and after the CUDA 13 PR, `cudaPackages.backendStdenv.cc` (`gcc-wrapper`) was leaking into the `firefox` output. However, before the CUDA 13 PR, `stdenv` and `cudaPackages.backendStdenv` were not the same. After the CUDA 13 PR, `stdenv` and `cudaPackages.backendStdenv` are the same. Hence, `disallowedRequisites = [ stdenv.cc ];` catches the `nvcc`-leaked `gcc-wrapper` (`cudaPackages.backendStdenv.cc`). So, who's fault is it? A) It is wrong that `stdenv == cudaPackages.backendStdenv`. Then the issue is not `cudaPackages.cuda_nvcc` leaking `gcc-wrapper`. B) It is normal that `stdenv == cudaPackages.backendStdenv`, but `cudaPackages.cuda_nvcc` should have never leaked `gcc-wrapper` in the first place.	21:21:05
Gaétan Lepage	* connor (burnt/out) (UTC-7) I found out the issue with `firefox`. Both prior and after the CUDA 13 PR, `cudaPackages.backendStdenv.cc` (`gcc-wrapper`) was leaking into the `firefox` output. However, before the CUDA 13 PR, `stdenv` and `cudaPackages.backendStdenv` were not the same. After the CUDA 13 PR, `stdenv` and `cudaPackages.backendStdenv` are the same. Hence, in `firefox` (wrapper) derivation, `disallowedRequisites = [ stdenv.cc ];` catches the `nvcc`-leaked `gcc-wrapper` (`cudaPackages.backendStdenv.cc`). So, who's fault is it? A) It is wrong that `stdenv == cudaPackages.backendStdenv`. Then the issue is not `cudaPackages.cuda_nvcc` leaking `gcc-wrapper`. B) It is normal that `stdenv == cudaPackages.backendStdenv`, but `cudaPackages.cuda_nvcc` should have never leaked `gcc-wrapper` in the first place.	21:21:43
apyh	In reply to @apyh:matrix.org omg. i wanna try. ripped it on your branch in 23m, including thr magma compile - only compute 8.9 tho	21:38:03
Gaétan Lepage	Oh, I was implying "all caps enabled"	21:39:18
apyh	lemme try it :3	21:43:58
connor (burnt/out) (UTC-8)	stdenv can be cudaPackages.backendStdenv if the version of GCC is supported by NVCC. It’s only different when we need to use an older version of GCC. NVCC shouldn’t leak the GCC wrapper since it should be largely build-time only. Any ideas why it’s propagating like that?	21:46:06
Gaétan Lepage	Thanks for the follow-up. The leakage chain is: `cudaPackages.cuda_nvcc` -> `cudaPackages.nccl` -> `firefox-unwrapped` -> `firefox`. In `cuda_nvcc.nix`, `backendStdenv.cc` is added into `cuda_nvcc`'s `propagatedBuildInputs`.	21:50:01
Gaétan Lepage	https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/cuda-modules/packages/cuda_nvcc.nix#L24-L25	21:50:06
apyh	do you have a command / nixpkgs config for me to work with or nah?	21:50:14
apyh	just wanna bench against yours :p	21:50:22
Gaétan Lepage	Actually, the leakage is not transitive. Firefox uses `disallowedRequisites`, not `disallowedReferences`. The former makes the specified references illegal in **all (transitive) dependencies. Conversely, `disallowedReferences` only forbids them in the result of the current derivation. Basically, I can build `cudaPackages.nccl` with `disallowedReferences = [ backendStdenv.cc]` bot not `disallowedRequisites = [ backendStdenv.cc ]`. So, if I understand correctly, having `propagatedBuildInputs = [ backendStdenv.cc ]` in `cuda_nvcc`'s derivation is fundamentally incompatible with having `disallowedRequisites` in `firefox`'s wrapper derivation.	22:13:01
Gaétan Lepage	So, either: A) we get rid of `propagatedBuildInputs = [ backendStdenv.cc ]` in `cuda_nvcc`'s derivation. But I suspect that it is there for a (good) reason. B) We relax `firefox`'s `disallowedRequisites` to `disallowedReferences`.	22:15:26
Gaétan Lepage	* Actually, the leakage is not transitive. Firefox uses `disallowedRequisites`, not `disallowedReferences`. The former makes the specified references illegal in all (transitive) dependencies. Conversely, `disallowedReferences` only forbids them in the result of the current derivation. Basically, I can build `cudaPackages.nccl` with `disallowedReferences = [ backendStdenv.cc]` bot not `disallowedRequisites = [ backendStdenv.cc ]`. So, if I understand correctly, having `propagatedBuildInputs = [ backendStdenv.cc ]` in `cuda_nvcc`'s derivation is fundamentally incompatible with having `disallowedRequisites` in `firefox`'s wrapper derivation.	22:17:36
Gaétan Lepage	Actually, I was able to build `python3Packages.torch` and `firefox` with an empty `propagatedBuildInputs` in `cuda_nvcc`. Why do we need it exactly?	23:25:46
connor (burnt/out) (UTC-8)	If it’s in propagatedBuildInputs it should still slide out of the dependencies far enough down It likely worked because the current stdenv is supported by the version of NVCC in the default CUDA package set stdenv.cc needs to be in NVCC’s propagatedBuildInputs because NVCC needs it available it when it is in nativeBuildInputs	23:28:42
connor (burnt/out) (UTC-8)	I believe Firefox-unwrapped and Firefox should use disallowedReferences. I’m trying to think why it’s okay to try to block that transitively?	23:30:54
Gaétan Lepage	Thanks! Opened https://github.com/NixOS/nixpkgs/pull/457391. Let's see what the firefox maintainers think.	23:42:18
1 Nov 2025
SomeoneSerge (back on matrix)	Catching up just now, but I do not see any conclusion on why and how firefox retains a path to gcc after the build?	01:50:57
SomeoneSerge (back on matrix)	That should not be happening	01:51:02
SomeoneSerge (back on matrix)	Ah. We didn't used to have that, cc in propagated inputs Since when do we? We did used to propagate a hook though, I forget under what conditions if any	01:53:05
SomeoneSerge (back on matrix)	`commit c03326445b067dca37ea323d998ffa3d520adb6d Author: Eelco Dolstra <edolstra@gmail.com> Date: Tue Sep 26 22:37:38 2017 +0200 firefox: Remove about:buildconfig Storing the build configuration caused Firefox to retain a dependency on gcc, glibc.dev and icu4c.dev. This reduces the size of the firefox closure from 587 to 415 MiB.` Wow that's old now	02:01:52
hexa (UTC+1)	And we still remove that, because it pulls in all kinds of shit and bloats the closure	02:02:13

Show newer messages

Back to Room ListRoom Version: 9