!eWOErHSaiddIbsUNsJ:nixos.org

NixOS CUDA

287 Members
CUDA packages maintenance and support in nixpkgs | https://github.com/orgs/NixOS/projects/27/ | https://nixos.org/manual/nixpkgs/unstable/#cuda58 Servers

You have reached the beginning of time (for this room).


SenderMessageTime
23 Sep 2025
@hugo:okeso.euHugoI also find it really weird that CPU stays at 100% while the build is stuck.10:22:25
@hugo:okeso.euHugoIs that a known bug in Nix or in the build tools?10:22:55
@a-kenji:matrix.orgkenji changed their display name from a-kenji to kenji.10:42:31
@albertlarsan68:albertlarsan.frAlbert Larsannvcc spawns multiple compiler instances per invocation, and ninja spawns as many nvcc instances as the number of cores/threads, which makes the CPU overcommited (ex: you have 16 threads, ninja spawn 16 nvcc instances, and each one of the nvcc intances spawns 6 cicc instances, and each cicc instance consumes one full cpu thread. So you end up with 16*6=96 processes trying to run at the same time). The build is not stuck, it just takes a very long time to happen (because it tries to do more at the same time than your computer can handle)11:32:01
@hugo:okeso.euHugoTanks for this explanation. My impression is more that the system goes OOM and then something gets stuck and never resumes.11:44:59
@hugo:okeso.euHugoEspecially since a working build (previous release) finished in 14 minutes.12:08:09
@gregorburger:matrix.orgGregor BurgerHi Guys, quick question is there an equivalent cudaPackages.backendStdenv for clang? 12:09:25
@gregorburger:matrix.orgGregor Burger* Hi, quick question is there an equivalent cudaPackages.backendStdenv for clang? 12:11:37
@connorbaker:matrix.orgconnor (burnt/out) (UTC-8)It looks like you’re out of memory and swapping hard; try lowering the number of cores given to the job or the number of parallel instances NVCC runs with (per what Albert said above). I’ve had to enable ZRAM (which has been highly effective) for some builds even on my desktops with 96GB of RAM.12:53:57
@hugo:okeso.euHugo

I have this policy on my server, and a similar one on my desktop. Should that not prevent the OS from swapping?

  systemd.services.nix-daemon.serviceConfig = {
    CPUAccounting = true;
    AllowedCPUs = "2-15";
    MemoryAccounting = true;
    MemoryHigh = "48G";
    MemoryMax = "56G";
  };
12:55:49
@connorbaker:matrix.orgconnor (burnt/out) (UTC-8)Currently no; I’m working on making the setup hooks and everything generic for Clang in https://github.com/NixOS/nixpkgs/pull/437723 but I ran into issues doing that and it’s not a high priority for me in the scope of that PR. Any particular use case?12:56:18
@connorbaker:matrix.orgconnor (burnt/out) (UTC-8)I don’t know enough about systemd to answer that, but I know some of the flash attention kernel builds consume at least a hundred GB of RAM and if you’re seeing the build stall that reminds me of the swapping to disk behavior I’d seen previously. (I may also have misinterpreted the BTOP screen shot.)12:58:25
@hugo:okeso.euHugo Apparently there is an extra setting MemorySwapMax = "0"; that can disallow swap for a systemd unit. 13:37:42

Show newer messages


Back to Room ListRoom Version: 9