How would you go about conditionally setting cudaCapabilities when instantiating nixpkgs? I.e.

Image I have this.

{
  inputs = {
    nixpkgs = "github:nixos/nixpkgs?ref=nixos-25.05";
  };
  
  outputs = { self, nixpkgs }: {
    packages.x86_64-linux.default = let
      pkgs = import nixpkgs {
        overlays = [ ];
        config = {
          allowUnfree = true;
          cudaSupport = true;
          cudaCapabilities = [ "..." "..." ];
        };
      };
    in
    pkgs.hello;
    
    packages.aarch64-linux.default = let
      pkgs = import nixpkgs {
        overlays = [ ];
        config = {
          allowUnfree = true;
          cudaSupport = true;
          cudaCapabilities = if isJetson then [ "..." "..." ] else [ "..." "..." ];
        };
      };
    in
    pkgs.hello;
  };
}

It's the aarch64-linux part specifically that I'm a bit stuck on. I have some cloud servers that have an NVIDIA GPUs in them that run aarch64-linux, but I also have some Jetson devices that are also considered aarch64-linux.

And if I understand the whole thing correctly, I can't just set the cudaCapabilities list to include both the non-jetson and jetson capabilities, right? Or at least, than isJetsonBuild would just always eval to true even if the build was meant for the cloud server.

Probably something stupid I'm just overlooking, sorry for bothering. 😅

It's the aarch64-linux part specifically that I'm a bit stuck

There's aarch64-linux and there's aarch64-linux. It's an artifact of us not including cuda/rocm stuff in hostPlatform (yet). The isJetsonBuild should only evaluate to true if your cudaCapabilities are jetson capabilities

In reply to @bjth:matrix.org

How would you go about conditionally setting cudaCapabilities when instantiating nixpkgs? I.e.

Image I have this.

{
  inputs = {
    nixpkgs = "github:nixos/nixpkgs?ref=nixos-25.05";
  };
  
  outputs = { self, nixpkgs }: {
    packages.x86_64-linux.default = let
      pkgs = import nixpkgs {
        overlays = [ ];
        config = {
          allowUnfree = true;
          cudaSupport = true;
          cudaCapabilities = [ "..." "..." ];
        };
      };
    in
    pkgs.hello;
    
    packages.aarch64-linux.default = let
      pkgs = import nixpkgs {
        overlays = [ ];
        config = {
          allowUnfree = true;
          cudaSupport = true;
          cudaCapabilities = if isJetson then [ "..." "..." ] else [ "..." "..." ];
        };
      };
    in
    pkgs.hello;
  };
}

It's the aarch64-linux part specifically that I'm a bit stuck on. I have some cloud servers that have an NVIDIA GPUs in them that run aarch64-linux, but I also have some Jetson devices that are also considered aarch64-linux.

And if I understand the whole thing correctly, I can't just set the cudaCapabilities list to include both the non-jetson and jetson capabilities, right? Or at least, than isJetsonBuild would just always eval to true even if the build was meant for the cloud server.

Probably something stupid I'm just overlooking, sorry for bothering. 😅

Aarch based nvidia data center gpus 👀, yeah if you get the correct map of the cuda capabilities it should work fine

Edit: misread, isJetsonBuild sounds funky so not sure

i'm confused about the compatibility story between whatever libcuda.so file i have in /run/opengl-driver and my nvidia kernel module. i've read through <nixos/modules/hardware/video/nvidia.nix> and i see that hardware.graphics.extraPackages basically gets set to pkgs.linuxKernel.packages.linux_6_12.nvidiaPackages.stable.out (or whatever kernel i have selected)

how much drift (if any) is allowed here?

In reply to @jfly:matrix.org
to avoid an XY problem: what i'm actually doing is experimenting with defining systemd nixos containers that run cuda software internally, and i'm not sure how to get the right libcuda.so's in those containers so they play nicely with the host's kernel

Do the systemd NixOS containers provide their own copy of NVIDIA's driver? If not, they wouldn't have libcuda.so available.

afaik, they do not automatically do anything (please correct me if i'm wrong). i making them get their own libcuda.so by explicitly configuring them with hardware.graphics.enable = true; and hardware.graphics.extraPackages.

mounting the cuda runtime from the host makes sense, though! thanks for the link to this nvidia-container-toolkit