Hi everyone! I am looking at a bug we have with CDI (Container Device Interface, for forwarding GPU's to containers): https://github.com/NixOS/nixpkgs/issues/397065

I think the user has a correct configuration (unless there are settings that were not mentioned in the issue), my main question is when using the datacenter driver, why the nvidia-container-toolkit is reporting:

ERRO[0000] failed to generate CDI spec: failed to create device CDI specs: failed to initialize NVML: ERROR_LIBRARY_NOT_FOUND

Do you have any idea on why NVML would not be present in this environment?

HI! I've a small announcement to make.

I've been failing badly to keep up with the backlog as a maintainer, even though I'm recently able to spend some more time on Nixpkgs&c. Working in occasional 1:1 meetings, otoh, has always felt comparatively productive. We've just had another call with Gaétan Lepage and I find it was nice, so I now want to try the following: https://md.someonex.net/s/9S4E00sIb#

This is not exactly "official", I'm not posting this e.g. on Discourse until I'm more confident, but as such it's an open invitation.

is this the right place to ask questions / get pointes on how to properly setup cuda container toolkit?

For docker it seems to work when enabling deprecated enableNvidia = true; flag. However with neither nvidia-container-toolkit in systemPackages with or without hardware.nvidia-container-toolkit.enable = true; I cannot seem to get it to run...

Should have read the docs properly. The relevant section section from

Did you manage to get containerd to work?

In reply to @luke-skywalker:matrix.org
ereslibre: I got it to run with docker but still struggling to getting it to run with containerd and k8s-device-plugin.

ereslibre: where exactly should I make an issue? on nixpkgs gituhb? If so how do I indicate that its about the nvidia-container-toolkit?

So far, the attached config got me to:

1. docker (+compose) runtime working with CUDA workloads
2. containerd runtime directly running cuda containers
3. get rke2 running with a config.toml that points to all needed runtimes in the nix store.

   hardware.nvidia-container-toolkit = {
      enable = true;
      # package = pkgs.nvidia-container-toolkit;
      # # Use UUID for device naming - better for multi-GPU setups
      device-name-strategy = "uuid"; # one of "index", "uuid", "type-index"
      # Mount additional directories for compatibility
      mount-nvidia-docker-1-directories = true;

      # Mount NVIDIA executables into container
      mount-nvidia-executables = true;
    };
    services.rke2 = {
      enable = true;
      role = "server";
      nodeName = "workstation-0";
      cni = "canal"; # | canal

      # Set the node IP directly
      nodeIP = "${systemProfile.network.staticIP}";
      debug = true;

      # Set cluster CIDR ranges properly
      extraFlags = [
        "--kubelet-arg=cgroup-driver=systemd"
        "--cluster-cidr=10.42.0.0/16"
        "--service-cidr=10.43.0.0/16"
        "--disable-cloud-controller" # Disable cloud controller for bare metal
        # "--kubelet-arg=feature-gates=DevicePlugins=true" # Add this for device plugins
      ];
      disable = ["traefik"]; # "servicelb"
      # environmentVars = {
      #   NVIDIA_VISIBLE_DEVICES = "all";
      #   NVIDIA_DRIVER_CAPABILITIES = "all";

      #   # Set NVIDIA driver root to the standard location
      #   # NVIDIA_DRIVER_ROOT = "/usr/lib/nvidia";

      #   # Home directory for RKE2
      #   HOME = "/root";
      # };
    };

/var/lib/rancher/rke2/agent/etc/containerd/config.toml