In reply to @ss:someonex.net
Jonas Chevalier hexa (UTC+1) a question about release-lib.nix: my impression is that supportedPlatforms is the conventional way to describe a "matrix" of jobs; for aarch64-linux, I'd like to define a matrix over individual capabilities because aarch64-linux mostly means embedded/jetson SBCs; currently this means importing nixpkgs with different config.cudaCapabilities values... any thoughts on how to express this in a not-too-ad-hoc way?

Patch release-lib.nix to add this logic:

nixpkgsArgs' = if builtins.isFunction nixpkgsArgs then nixpkgsArgs else (system: nixpkgsArgs);

And then replace all the nixpkgsArgs usages with (nixpkgsArgs' system)

[2/2137/2141 built (157 failed), 11826 copied (948537.2 MiB), 27313.5 MiB DL] building xyce-7.8.0 (checkPhase): MEASURE/PrecisionTest............................................passed[sh]      (Time:   1s =   0.

💀

This might just be me being dumb, but am surprised that I'm unable to build jax with doCheck = false (use case is I want to override jaxlib = jaxlibWithCuda and don't want to run the tests). Repro:

nix build --impure --expr 'let nixpkgs = builtins.getFlake("github:nixos/nixpkgs/nixpkgs-unstable"); pkgs = import nixpkgs { system = "x86_64-linux"; config.allowUnfree = true; }; in pkgs.python3Packages.jax.overridePythonAttrs { doCheck = false; }'

fails with ModuleNotFoundError: jax requires jaxlib to be installed

Something else that's puzzling me: overriding with jaxlib = jaxlibWithCuda doesn't seem to work for numpyro (which has a pretty simple derivation):

Sanity check: a Python env with just jax and jaxlibWithCuda is GPU-enabled, as expected:

nix shell --refresh --impure --expr 'let nixpkgs = builtins.getFlake "github:nixos/nixpkgs/nixpkgs-unstable"; pkgs = import nixpkgs { config.allowUnfree = true; }; in (pkgs.python3.withPackages (ps: [ ps.jax ps.jaxlibWithCuda ]))' --command python -c "import jax; print(jax.devices())"

yields [cuda(id=0), cuda(id=1)].

But when numpyro is overridden to use jaxlibWithCuda, for some reason the propagated jaxlib is still the CPU version:

nix shell --refresh --impure --expr 'let nixpkgs = builtins.getFlake "github:nixos/nixpkgs/nixpkgs-unstable"; pkgs = import nixpkgs { config.allowUnfree = true; }; in (pkgs.python3.withPackages (ps: [ ((ps.numpyro.overridePythonAttrs (_: { doCheck = false; })).override { jaxlib = ps.jaxlibWithCuda; }) ]))' --command python -c "import jax; print(jax.devices())"

yields [CpuDevice(id=0)]. (Furthermore, if we try to add jaxlibWithCuda to the withPackages call, we get a collision error, so clearly something is propagating the CPU jaxlib 🤔

Has anyone seen this, or have a better way to use the GPU-enabled jaxlib as a dependency?

Something else that's puzzling me: overriding with jaxlib = jaxlibWithCuda doesn't seem to work for numpyro (which has a pretty simple derivation):

Sanity check: a Python env with just jax and jaxlibWithCuda is GPU-enabled, as expected:

nix shell --refresh --impure --expr 'let nixpkgs = builtins.getFlake "github:nixos/nixpkgs/nixpkgs-unstable"; pkgs = import nixpkgs { config.allowUnfree = true; }; in (pkgs.python3.withPackages (ps: [ ps.jax ps.jaxlibWithCuda ]))' --command python -c "import jax; print(jax.devices())"

yields [cuda(id=0), cuda(id=1)].

But when numpyro is overridden to use jaxlibWithCuda, for some reason the propagated jaxlib is still the CPU version:

nix shell --refresh --impure --expr 'let nixpkgs = builtins.getFlake "github:nixos/nixpkgs/nixpkgs-unstable"; pkgs = import nixpkgs { config.allowUnfree = true; }; in (pkgs.python3.withPackages (ps: [ ((ps.numpyro.overridePythonAttrs (_: { doCheck = false; })).override { jaxlib = ps.jaxlibWithCuda; }) ]))' --command python -c "import jax; print(jax.devices())"

yields [CpuDevice(id=0)]. (Furthermore, if we try to add jaxlibWithCuda to the withPackages call, we get a collision error, so clearly something is propagating the CPU jaxlib 🤔)

Has anyone seen this, or have a better way to use the GPU-enabled jaxlib as a dependency?

In reply to @mcwitt:matrix.org

Something else that's puzzling me: overriding with jaxlib = jaxlibWithCuda doesn't seem to work for numpyro (which has a pretty simple derivation):

Sanity check: a Python env with just jax and jaxlibWithCuda is GPU-enabled, as expected:

nix shell --refresh --impure --expr 'let nixpkgs = builtins.getFlake "github:nixos/nixpkgs/nixpkgs-unstable"; pkgs = import nixpkgs { config.allowUnfree = true; }; in (pkgs.python3.withPackages (ps: [ ps.jax ps.jaxlibWithCuda ]))' --command python -c "import jax; print(jax.devices())"

yields [cuda(id=0), cuda(id=1)].

But when numpyro is overridden to use jaxlibWithCuda, for some reason the propagated jaxlib is still the CPU version:

nix shell --refresh --impure --expr 'let nixpkgs = builtins.getFlake "github:nixos/nixpkgs/nixpkgs-unstable"; pkgs = import nixpkgs { config.allowUnfree = true; }; in (pkgs.python3.withPackages (ps: [ ((ps.numpyro.overridePythonAttrs (_: { doCheck = false; })).override { jaxlib = ps.jaxlibWithCuda; }) ]))' --command python -c "import jax; print(jax.devices())"

yields [CpuDevice(id=0)]. (Furthermore, if we try to add jaxlibWithCuda to the withPackages call, we get a collision error, so clearly something is propagating the CPU jaxlib 🤔)

Has anyone seen this, or have a better way to use the GPU-enabled jaxlib as a dependency?

In reply to @connorbaker:matrix.org
Were there any changes to Python packages over the last month or so that may have caused breakages? Like default version or anything? I rebased a PR and I’m seeing more build failures than I was previously

Trying to post the results of a nixpkgs-review run and:

There was a problem saving your comment. Your comment is too long (maximum is 65536 characters). Please try again.

SomeoneSerge (UTC+3): I'm having a bit of trouble with GPU-access in the sandbox. In particular, I've enabled it in my NixOS config with

{
  programs.nix-required-mounts = {
    enable = true;
    presets.nvidia-gpu.enable = true;
    # TODO: Fix merging of presets
    # error: The option `programs.nix-required-mounts.allowedPatterns.nvidia-gpu.unsafeFollowSymlinks' has conflicting definition values:
    # - In `/nix/store/dk2rpyb6ndvfbf19bkb2plcz5y3k8i5v-source/nixos/modules/programs/nix-required-mounts.nix': false
    # - In `/nix/store/2ja2h1nd0z2bw56cl4bn37cb9d18hnzr-source/devices/nixos-desktop/hardware.nix': true
    # Use `lib.mkForce value` or `lib.mkDefault value` to change the priority on any of these definitions.
    # TODO: After enabling running into this error when trying to build a derivation which has requiredSystemFeatures = [ "cuda" ];
    # error:
    #   … while setting up the build environment
    #
    #   error: getting attributes of path '/nix/store/0p1qsszik7hwjddzmyhikq9ywr2ki69l-systemd-minimal-255.6/sbin/bin': No such file or directory
    # Perhaps this is due to the systemd directory in /run/opengl-driver/lib?
    allowedPatterns.nvidia-gpu.unsafeFollowSymlinks = lib.mkForce true;
  };
}

I'm trying to enable the GPU portion of CMake's CUDA test suite (https://github.com/ConnorBaker/nixpkgs/commit/543cf7d2ec330286ba566e6e1187e531d155c5d0), but failing. I thought following the symlinks when mounting would help because before that I was unable to access the ${addDriverRunpath.driverLink}/lib directory (multiple symlinks), but it now fails with

$ nix build --impure -L .#cudaPackages.cmake-cuda-tests.tests.withGpu
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
warning: killing stray builder process 5264 ()...
error:
       … while setting up the build environment

       error: getting attributes of path '/nix/store/0p1qsszik7hwjddzmyhikq9ywr2ki69l-systemd-minimal-255.6/sbin/bin': No such file or directory

Any ideas?

Ah, okay.
The think addDriverRunpath.driverLink links to is /run/opengl-driver. That is in turn a symlink, created by this: https://github.com/NixOS/nixpkgs/blob/c82d9d313d5107c6ad3a92fc7d20343f45fa5ace/nixos/modules/hardware/graphics.nix#L5-L8
That derivation isn't expose except as a path, used here:
https://github.com/NixOS/nixpkgs/blob/c82d9d313d5107c6ad3a92fc7d20343f45fa5ace/nixos/modules/hardware/graphics.nix#L112-L121
I updated my nixos config as follows, and it seems to work.

{
  programs.nix-required-mounts = {
    enable = true;
    presets.nvidia-gpu.enable = true;
    allowedPatterns.nvidia-gpu = {
      onFeatures = [
        "gpu"
        "nvidia-gpu"
        "opengl"
        "cuda"
      ];
      # It exposes these paths in the sandbox:
      paths =
        let
          inherit (pkgs.addOpenGLRunpath) driverLink;
          thingDriverLinkLinksTo =
            config.systemd.tmpfiles.settings.graphics-driver."/run/opengl-driver"."L+".argument;
        in
        [
          driverLink
          thingDriverLinkLinksTo
          "/dev/dri"
          "/dev/nvidia*"
        ];
    };
  };
}

Of course, that same process would need to be repeated for anything in there which is in turn a symlink (which is the purpose of unsafeFollowSymlinks, I suppose), but I'm not getting that odd systemd bin error any more.