Hi everyone! I am looking at a bug we have with CDI (Container Device Interface, for forwarding GPU's to containers): https://github.com/NixOS/nixpkgs/issues/397065

I think the user has a correct configuration (unless there are settings that were not mentioned in the issue), my main question is when using the datacenter driver, why the nvidia-container-toolkit is reporting:

ERRO[0000] failed to generate CDI spec: failed to create device CDI specs: failed to initialize NVML: ERROR_LIBRARY_NOT_FOUND

Do you have any idea on why NVML would not be present in this environment?

HI! I've a small announcement to make.

I've been failing badly to keep up with the backlog as a maintainer, even though I'm recently able to spend some more time on Nixpkgs&c. Working in occasional 1:1 meetings, otoh, has always felt comparatively productive. We've just had another call with Gaétan Lepage and I find it was nice, so I now want to try the following: https://md.someonex.net/s/9S4E00sIb#

This is not exactly "official", I'm not posting this e.g. on Discourse until I'm more confident, but as such it's an open invitation.

is this the right place to ask questions / get pointes on how to properly setup cuda container toolkit?

For docker it seems to work when enabling deprecated enableNvidia = true; flag. However with neither nvidia-container-toolkit in systemPackages with or without hardware.nvidia-container-toolkit.enable = true; I cannot seem to get it to run...

Should have read the docs properly. The relevant section section from

Did you manage to get containerd to work?

In reply to @luke-skywalker:matrix.org
ereslibre: I got it to run with docker but still struggling to getting it to run with containerd and k8s-device-plugin.

ereslibre: where exactly should I make an issue? on nixpkgs gituhb? If so how do I indicate that its about the nvidia-container-toolkit?

So far, the attached config got me to:

1. docker (+compose) runtime working with CUDA workloads
2. containerd runtime directly running cuda containers
3. get rke2 running with a config.toml that points to all needed runtimes in the nix store.

   hardware.nvidia-container-toolkit = {
      enable = true;
      # package = pkgs.nvidia-container-toolkit;
      # # Use UUID for device naming - better for multi-GPU setups
      device-name-strategy = "uuid"; # one of "index", "uuid", "type-index"
      # Mount additional directories for compatibility
      mount-nvidia-docker-1-directories = true;

      # Mount NVIDIA executables into container
      mount-nvidia-executables = true;
    };
    services.rke2 = {
      enable = true;
      role = "server";
      nodeName = "workstation-0";
      cni = "canal"; # | canal

      # Set the node IP directly
      nodeIP = "${systemProfile.network.staticIP}";
      debug = true;

      # Set cluster CIDR ranges properly
      extraFlags = [
        "--kubelet-arg=cgroup-driver=systemd"
        "--cluster-cidr=10.42.0.0/16"
        "--service-cidr=10.43.0.0/16"
        "--disable-cloud-controller" # Disable cloud controller for bare metal
        # "--kubelet-arg=feature-gates=DevicePlugins=true" # Add this for device plugins
      ];
      disable = ["traefik"]; # "servicelb"
      # environmentVars = {
      #   NVIDIA_VISIBLE_DEVICES = "all";
      #   NVIDIA_DRIVER_CAPABILITIES = "all";

      #   # Set NVIDIA driver root to the standard location
      #   # NVIDIA_DRIVER_ROOT = "/usr/lib/nvidia";

      #   # Home directory for RKE2
      #   HOME = "/root";
      # };
    };

/var/lib/rancher/rke2/agent/etc/containerd/config.toml

ereslibre: where exactly should I make an issue? on nixpkgs gituhb? If so how do I indicate that its about the nvidia-container-toolkit?

So far, the attached config got me to:

1. docker (+compose) runtime working with CUDA workloads
2. containerd runtime directly running cuda containers
3. get rke2 running with a config.toml that points to all needed runtimes in the nix store.

   hardware.nvidia-container-toolkit = {
      enable = true;
      # package = pkgs.nvidia-container-toolkit;
      # # Use UUID for device naming - better for multi-GPU setups
      device-name-strategy = "uuid"; # one of "index", "uuid", "type-index"
      # Mount additional directories for compatibility
      mount-nvidia-docker-1-directories = true;

      # Mount NVIDIA executables into container
      mount-nvidia-executables = true;
    };
    services.rke2 = {
      enable = true;
      role = "server";
      nodeName = "workstation-0";
      cni = "canal"; # | canal

      # Set the node IP directly
      nodeIP = "${systemProfile.network.staticIP}";
      debug = true;

      # Set cluster CIDR ranges properly
      extraFlags = [
        "--kubelet-arg=cgroup-driver=systemd"
        "--cluster-cidr=10.42.0.0/16"
        "--service-cidr=10.43.0.0/16"
        "--disable-cloud-controller" # Disable cloud controller for bare metal
        # "--kubelet-arg=feature-gates=DevicePlugins=true" # Add this for device plugins
      ];
      disable = ["traefik"]; # "servicelb"
      # environmentVars = {
      #   NVIDIA_VISIBLE_DEVICES = "all";
      #   NVIDIA_DRIVER_CAPABILITIES = "all";

      #   # Set NVIDIA driver root to the standard location
      #   # NVIDIA_DRIVER_ROOT = "/usr/lib/nvidia";

      #   # Home directory for RKE2
      #   HOME = "/root";
      # };
    };

/var/lib/rancher/rke2/agent/etc/containerd/config.toml

# File generated by rke2. DO NOT EDIT. Use config.toml.tmpl instead.
version = 3
root = "/var/lib/rancher/rke2/agent/containerd"
state = "/run/k3s/containerd"

[grpc]
  address = "/run/k3s/containerd/containerd.sock"

[plugins.'io.containerd.internal.v1.opt']
  path = "/var/lib/rancher/rke2/agent/containerd"

[plugins.'io.containerd.grpc.v1.cri']
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"

[plugins.'io.containerd.cri.v1.runtime']
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  device_ownership_from_security_context = false

[plugins.'io.containerd.cri.v1.images']
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true

[plugins.'io.containerd.cri.v1.images'.pinned_images]
  sandbox = "index.docker.io/rancher/mirrored-pause:3.6"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
  SystemdCgroup = true

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runhcs-wcow-process]
  runtime_type = "io.containerd.runhcs.v1"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia']
  runtime_type = "io.containerd.runc.v2"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia'.options]
  BinaryName = "/var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/nvidia-container-runtime"
  SystemdCgroup = true

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia-cdi']
  runtime_type = "io.containerd.runc.v2"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia-cdi'.options]
  BinaryName = "/var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/nvidia-container-runtime.cdi"
  SystemdCgroup = true

[plugins.'io.containerd.cri.v1.images'.registry]
  config_path = "/var/lib/rancher/rke2/agent/etc/containerd/certs.d"

However when trying to deploy the nvidia device plugin either with rke2 operator or as simple daemonset or as helm chart from the nvidia-device-plugin repo, it fails on detecing the cuda environment. for example by complaining about auto strategy.