| 28 Jun 2024 |
 Titus | File "/home/titus/src/bnb/bitsandbytes/diagnostics/main.py", line 66, in main
sanity_check()
File "/home/titus/src/bnb/bitsandbytes/diagnostics/main.py", line 33, in sanity_check
p = torch.nn.Parameter(torch.rand(10, 10).cuda())
File "/home/titus/src/bnb/.pixi/envs/default/lib/python3.8/site-packages/torch/cuda/__init__.py", line 293, in _lazy_init
torch._C._cuda_init()
| 13:06:53 |
 SomeoneSerge (matrix works sometimes) |
pixi
Is one of the package managers that ship incomplete dependencies because they expect an FHS environment. If you want to use it on NixOS I recommend you use nix-ld. You'll aslo need to ensure (using a shell e..g) that the ld.so is aware of ${addDriverRunpath.driverLink}/lib, which you can also do as part of the nix-ld configuration. E.g. you can deploy a nixos with programs.nix-ld.enable and then, in your project tree, use a nix shell that looks something like the following: https://github.com/NixOS/nixpkgs/blob/48dbb2ae90be0ba21b44e77b8278fd7cefb4b75f/nixos/doc/manual/configuration/fhs.chapter.md?plain=1#L105-L113 | 13:09:09 |
| SomeoneSerge (matrix works sometimes) | pkgs.buildFHSUserEnv {
name = "pixi-env";
targetPkgs = pkgs: with pkgs; [
...
nvidia_x11
...
];
...this, if it had any affect on the dynamic loader (in this form it doesn't, instead it provides hints for the compiler), would conflict with the libcuda driver deployed by NixOS. NVidia makes it so that the driver has to be deployed impurely, because each libcuda only works with the corresponding kernel. TLDR: delete nvidia_x11 from that list | 13:12:14 |
| SomeoneSerge (matrix works sometimes) | Also note that cudaPackages.cudatoolkit is a package for development; e.g. if pixi runs any builds (idk if it does) and you want it to use nixpkgs' cudatoolkit libraries instead of pixi libraries, that's when you include it int he shell | 13:13:27 |
| SomeoneSerge (matrix works sometimes) | * Also note that cudaPackages.cudatoolkit is a package for development; e.g. if pixi runs any builds (idk if it does) and you want it to use nixpkgs' cudatoolkit libraries instead of pixi libraries, that's when you include it in the shell | 13:13:30 |
| Titus | Thank you so much for your help, SO appreciated!
yeah, I had added nvidia_x11 only in the latest iteration.. I'll remove it right away.
The weird thing is that even outside the fhs, nvidia smi gives me this N/A: Have you ever seen that before? Does it tell us sth useful?
❯ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Aug_15_22:02:13_PDT_2023
Cuda compilation tools, release 12.2, V12.2.140
Build cuda_12.2.r12.2/compiler.33191640_0
| 13:40:43 |
| Titus | * Thank you so much for your help, SO appreciated!
yeah, I had added nvidia_x11 only in the latest iteration.. I'll remove it right away.
The weird thing is that even outside the fhs, nvidia smi gives me this N/A: Have you ever seen that before? Does it tell us sth useful?
❯ nvidia-smi
Fri Jun 28 14:55:14 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.78 Driver Version: 550.78 CUDA Version: ERR! |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4090 On | 00000000:01:00.0 On | Off |
| 0% 43C P8 40W / 480W | 3302MiB / 24564MiB | 26% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 4090 On | 00000000:03:00.0 Off | Off |
| 0% 36C P8 29W / 480W | 12MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 13:41:09 |
| SomeoneSerge (matrix works sometimes) | In reply to @titus-von-koeller:matrix.org
Thank you so much for your help, SO appreciated!
yeah, I had added nvidia_x11 only in the latest iteration.. I'll remove it right away.
The weird thing is that even outside the fhs, nvidia smi gives me this N/A: Have you ever seen that before? Does it tell us sth useful?
❯ nvidia-smi
Fri Jun 28 14:55:14 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.78 Driver Version: 550.78 CUDA Version: ERR! |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4090 On | 00000000:01:00.0 On | Off |
| 0% 43C P8 40W / 480W | 3302MiB / 24564MiB | 26% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 4090 On | 00000000:03:00.0 Off | Off |
| 0% 36C P8 29W / 480W | 12MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
That is odd. Just to make sure, you did reboot after you had set config.boot.kernelPackages.nvidiaPackages.beta and enabled hardware.opengl? If not, please reboot because these drivers are an edge case where mutable state matters... | 13:51:30 |
| SomeoneSerge (matrix works sometimes) | If you did, could post the output of nix run -f '<nixpkgs>' cudaPackages.saxpy? | 13:52:18 |
| Titus | ❯ nix run -f '<nixpkgs>' cudaPackages.saxpy
trace: warning: cudaPackages.autoAddDriverRunpath is deprecated, use pkgs.autoAddDriverRunpath instead
Start
Runtime version: 12020
Driver version: 0
Host memory initialized, copying to the device
CUDA error at cudaMalloc(&xDevice, N * sizeof(float)): CUDA driver is a stub library
| 14:47:01 |
| Titus | ok, so I updated my pixi-fhs.nix according to my best understanding of what you proposed, unfortunately, the error is still the exact same.
Would you mind taking another look? I also added some other stuff to give a better overview, please let me know if any other info is needed..
https://gist.github.com/Titus-von-Koeller/e541c1175b0a191bac75b72d9964e9d0 | 15:10:22 |
| SomeoneSerge (matrix works sometimes) | Ls /run/opengl-driver/lib/libcuda* | 15:10:39 |
| SomeoneSerge (matrix works sometimes) | Ican habea look tmr, nowI'm just typing from the phone... | 15:11:13 |
| Titus | ❯ ls /run/opengl-driver/lib/libcuda*
/run/opengl-driver/lib/libcuda.so -> /nix/store/2paf3i2g4arx5j4m9l87zdrzsikwmizh-nvidia-x11-550.78-6.6.35/lib/libcuda.so*
/run/opengl-driver/lib/libcuda.so.1 -> /nix/store/2paf3i2g4arx5j4m9l87zdrzsikwmizh-nvidia-x11-550.78-6.6.35/lib/libcuda.so.1*
/run/opengl-driver/lib/libcuda.so.550.78 -> /nix/store/2paf3i2g4arx5j4m9l87zdrzsikwmizh-nvidia-x11-550.78-6.6.35/lib/libcuda.so.550.78*
/run/opengl-driver/lib/libcudadebugger.so -> /nix/store/2paf3i2g4arx5j4m9l87zdrzsikwmizh-nvidia-x11-550.78-6.6.35/lib/libcudadebugger.so*
/run/opengl-driver/lib/libcudadebugger.so.1 -> /nix/store/2paf3i2g4arx5j4m9l87zdrzsikwmizh-nvidia-x11-550.78-6.6.35/lib/libcudadebugger.so.1*
/run/opengl-driver/lib/libcudadebugger.so.550.78 -> /nix/store/2paf3i2g4arx5j4m9l87zdrzsikwmizh-nvidia-x11-550.78-6.6.35/lib/libcudadebugger.so.550.78*
| 15:11:19 |
| SomeoneSerge (matrix works sometimes) | In reply to @titus-von-koeller:matrix.org
❯ nix run -f '<nixpkgs>' cudaPackages.saxpy
trace: warning: cudaPackages.autoAddDriverRunpath is deprecated, use pkgs.autoAddDriverRunpath instead
Start
Runtime version: 12020
Driver version: 0
Host memory initialized, copying to the device
CUDA error at cudaMalloc(&xDevice, N * sizeof(float)): CUDA driver is a stub library
Ok replace run with shell | 15:11:44 |
| SomeoneSerge (matrix works sometimes) | Thein the shell do
LD_ DEBUG=libs saxpy | 15:12:15 |
| Titus | Redacted or Malformed Event | 15:12:20 |
| Titus | Redacted or Malformed Event | 15:12:22 |
| SomeoneSerge (matrix works sometimes) | * Then in the shell do
LD_ DEBUG=libs saxpy
| 15:12:23 |
| Titus | Redacted or Malformed Event | 15:12:42 |
| Titus | ❯ LD_DEBUG=libs saxpy | xclip -sel clip
31778: find library=libcublas.so.12 [0]; searching
31778: search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/glibc-hwcaps/x86-64-v4:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/glibc-hwcaps/x86-64-v3:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/glibc-hwcaps/x86-64-v2:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v4:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v3:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v2:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v4:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v3:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v2:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs (LD_LIBRARY_PATH)
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/glibc-hwcaps/x86-64-v4/libcublas.so.12
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/glibc-hwcaps/x86-64-v3/libcublas.so.12
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/glibc-hwcaps/x86-64-v2/libcublas.so.12
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcublas.so.12
31778:
31778: find library=libcudart.so.12 [0]; searching
31778: search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v4:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v3:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v2:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v4:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v3:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v2:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs (LD_LIBRARY_PATH)
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcudart.so.12
31778:
31778: find library=libcublasLt.so.12 [0]; searching
31778: search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v4:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v3:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v2:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v4:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v3:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v2:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs (LD_LIBRARY_PATH)
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcublasLt.so.12
31778:
31778: find library=libdl.so.2 [0]; searching
31778: search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v4:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v3:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v2:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v4:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v3:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v2:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs (LD_LIBRARY_PATH)
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libdl.so.2
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v4/libdl.so.2
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v3/libdl.so.2
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v2/libdl.so.2
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/libdl.so.2
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v4/libdl.so.2
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v3/libdl.so.2
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v2/libdl.so.2
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/libdl.so.2
31778: search path=/run/opengl-driver/lib/glibc-hwcaps/x86-64-v4:/run/opengl-driver/lib/glibc-hwcaps/x86-64-v3:/run/opengl-driver/lib/glibc-hwcaps/x86-64-v2:/run/opengl-driver/lib (RUNPATH from file saxpy)
31778: trying file=/run/opengl-driver/lib/glibc-hwcaps/x86-64-v4/libdl.so.2
31778: trying file=/run/opengl-driver/lib/glibc-hwcaps/x86-64-v3/libdl.so.2
31778: trying file=/run/opengl-driver/lib/glibc-hwcaps/x86-64-v2/libdl.so.2
31778: trying file=/run/opengl-driver/lib/libdl.so.2
31778: search path=/nix/store/qdsimqhpg5ln8fy3xnaywrqdqsp7sglh-glib-2.80.2-bin/lib/glibc-hwcaps/x86-64-v4:/nix/store/qdsimqhpg5ln8fy3xnaywrqdqsp7sglh-glib-2.80.2-bin/lib/glibc-hwcaps/x86-64-v3:/nix/store/qdsimqhpg5ln8fy3xnaywrqdqsp7sglh-glib-2.80.2-bin/lib/glibc-hwcaps/x86-64-v2:/nix/store/qdsimqhpg5ln8fy3xnaywrqdqsp7sglh-glib-2.80.2-bin/lib:/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/glibc-hwcaps/x86-64-v4:/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/glibc-hwcaps/x86-64-v3:/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/glibc-hwcaps/x86-64-v2:/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib (LD_LIBRARY_PATH)
31778: trying file=/nix/store/qdsimqhpg5ln8fy3xnaywrqdqsp7sglh-glib-2.80.2-bin/lib/glibc-hwcaps/x86-64-v4/libdl.so.2
31778: trying file=/nix/store/qdsimqhpg5ln8fy3xnaywrqdqsp7sglh-glib-2.80.2-bin/lib/glibc-hwcaps/x86-64-v3/libdl.so.2
31778: trying file=/nix/store/qdsimqhpg5ln8fy3xnaywrqdqsp7sglh-glib-2.80.2-bin/lib/glibc-hwcaps/x86-64-v2/libdl.so.2
31778: trying file=/nix/store/qdsimqhpg5ln8fy3xnaywrqdqsp7sglh-glib-2.80.2-bin/lib/libdl.so.2
31778: trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/glibc-hwcaps/x86-64-v4/libdl.so.2
31778: trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/glibc-hwcaps/x86-64-v3/libdl.so.2
31778: trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/glibc-hwcaps/x86-64-v2/libdl.so.2
31778: trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/libdl.so.2
31778: search path=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/glibc-hwcaps/x86-64-v4:/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/glibc-hwcaps/x86-64-v3:/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/glibc-hwcaps/x86-64-v2:/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib (RUNPATH from file saxpy)
31778: trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/glibc-hwcaps/x86-64-v4/libdl.so.2
31778: trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/glibc-hwcaps/x86-64-v3/libdl.so.2
31778: trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/glibc-hwcaps/x86-64-v2/libdl.so.2
31778: trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libdl.so.2
31778: search path=/run/opengl-driver-32/lib/glibc-hwcaps/x86-64-v4:/run/opengl-driver-32/lib/glibc-hwcaps/x86-64-v3:/run/opengl-driver-32/lib/glibc-hwcaps/x86-64-v2:/run/opengl-driver-32/lib (LD_LIBRARY_PATH)
31778: trying file=/run/opengl-driver-32/lib/glibc-hwcaps/x86-64-v4/libdl.so.2
31778: trying file=/run/opengl-driver-32/lib/glibc-hwcaps/x86-64-v3/libdl.so.2
31778: trying file=/run/opengl-driver-32/lib/glibc-hwcaps/x86-64-v2/libdl.so.2
31778: trying file=/run/opengl-driver-32/lib/libdl.so.2
31778: search path=/run/opengl-driver/lib:/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/glibc-hwcaps/x86-64-v4:/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/glibc-hwcaps/x86-64-v3:/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/glibc-hwcaps/x86-64-v2:/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib:/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/glibc-hwcaps/x86-64-v4:/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/glibc-hwcaps/x86-64-v3:/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/glibc-hwcaps/x86-64-v2:/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib (RUNPATH from file saxpy)
31778: trying file=/run/opengl-driver/lib/libdl.so.2
31778: trying file=/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/glibc-hwcaps/x86-64-v4/libdl.so.2
31778: trying file=/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/glibc-hwcaps/x86-64-v3/libdl.so.2
31778: trying file=/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/glibc-hwcaps/x86-64-v2/libdl.so.2
31778: trying file=/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/libdl.so.2
31778: trying file=/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/glibc-hwcaps/x86-64-v4/libdl.so.2
31778: trying file=/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/glibc-hwcaps/x86-64-v3/libdl.so.2
31778: trying file=/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/glibc-hwcaps/x86-64-v2/libdl.so.2
31778: trying file=/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/libdl.so.2
31778: search path=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/glibc-hwcaps/x86-64-v4:/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/glibc-hwcaps/x86-64-v3:/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/glibc-hwcaps/x86-64-v2:/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib (system search path)
31778: trying file=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/glibc-hwcaps/x86-64-v4/libdl.so.2
31778: trying file=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/glibc-hwcaps/x86-64-v3/libdl.so.2
31778: trying file=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/glibc-hwcaps/x86-64-v2/libdl.so.2
31778: trying file=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libdl.so.2
31778:
31778: find library=librt.so.1 [0]; searching
31778: search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib (LD_LIBRARY_PATH)
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/librt.so.1
31778: search path=/run/opengl-driver/lib (RUNPATH from file saxpy)
31778: trying file=/run/opengl-driver/lib/librt.so.1
31778: search path=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib (LD_LIBRARY_PATH)
31778: trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/librt.so.1
31778: search path=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib (RUNPATH from file saxpy)
31778: trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/librt.so.1
31778: search path=/run/opengl-driver-32/lib (LD_LIBRARY_PATH)
31778: trying file=/run/opengl-driver-32/lib/librt.so.1
31778: search path=/run/opengl-driver/lib:/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib:/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib (RUNPATH from file saxpy)
31778: trying file=/run/opengl-driver/lib/librt.so.1
31778: trying file=/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/librt.so.1
31778: trying file=/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/librt.so.1
31778: search path=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib (system search path)
31778: trying file=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/librt.so.1
31778:
31778: find library=libpthread.so.0 [0]; searching
31778: search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib (LD_LIBRARY_PATH)
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libpthread.so.0
31778: search path=/run/opengl-driver/lib (RUNPATH from file saxpy)
31778: trying file=/run/opengl-driver/lib/libpthread.so.0
31778: search path=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib (LD_LIBRARY_PATH)
31778: trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/libpthread.so.0
31778: search path=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib (RUNPATH from file saxpy)
31778: trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libpthread.so.0
31778: search path=/run/opengl-driver-32/lib (LD_LIBRARY_PATH)
31778: trying file=/run/opengl-driver-32/lib/libpthread.so.0
31778: search path=/run/opengl-driver/lib:/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib:/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib (RUNPATH from file saxpy)
31778: trying file=/run/opengl-driver/lib/libpthread.so.0
31778: trying file=/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/libpthread.so.0
31778: trying file=/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/libpthread.so.0
31778: search path=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib (system search path)
31778: trying file=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libpthread.so.0
31778:
31778: find library=libstdc++.so.6 [0]; searching
31778: search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib (LD_LIBRARY_PATH)
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libstdc++.so.6
31778: search path=/run/opengl-driver/lib (RUNPATH from file saxpy)
31778: trying file=/run/opengl-driver/lib/libstdc++.so.6
31778: search path=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib (LD_LIBRARY_PATH)
31778: trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/libstdc++.so.6
31778: search path=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib (RUNPATH from file saxpy)
31778: trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
31778:
31778: find library=libm.so.6 [0]; searching
31778: search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib (LD_LIBRARY_PATH)
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libm.so.6
31778: search path=/run/opengl-driver/lib (RUNPATH from file saxpy)
31778: trying file=/run/opengl-driver/lib/libm.so.6
31778: search path=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib (LD_LIBRARY_PATH)
31778: trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/libm.so.6
31778: search path=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib (RUNPATH from file saxpy)
31778: trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libm.so.6
31778: search path=/run/opengl-driver-32/lib (LD_LIBRARY_PATH)
31778: trying file=/run/opengl-driver-32/lib/libm.so.6
31778: search path=/run/opengl-driver/lib:/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib:/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib (RUNPATH from file saxpy)
31778: trying file=/run/opengl-driver/lib/libm.so.6
31778: trying file=/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/libm.so.6
31778: trying file=/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/libm.so.6
31778: search path=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib (system search path)
31778: trying file=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libm.so.6
31778:
31778: find library=libgcc_s.so.1 [0]; searching
31778: search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib (LD_LIBRARY_PATH)
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libgcc_s.so.1
31778: search path=/run/opengl-driver/lib (RUNPATH from file saxpy)
31778: trying file=/run/opengl-driver/lib/libgcc_s.so.1
31778: search path=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib (LD_LIBRARY_PATH)
31778: trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/libgcc_s.so.1
31778: search path=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib (RUNPATH from file saxpy)
31778: trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libgcc_s.so.1
31778:
31778: find library=libc.so.6 [0]; searching
31778: search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib (LD_LIBRARY_PATH)
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libc.so.6
31778: search path=/run/opengl-driver/lib (RUNPATH from file saxpy)
31778: trying file=/run/opengl-driver/lib/libc.so.6
31778: search path=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib (LD_LIBRARY_PATH)
31778: trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/libc.so.6
31778: search path=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib (RUNPATH from file saxpy)
31778: trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libc.so.6
31778: search path=/run/opengl-driver-32/lib (LD_LIBRARY_PATH)
31778: trying file=/run/opengl-driver-32/lib/libc.so.6
31778: search path=/run/opengl-driver/lib:/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib:/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib (RUNPATH from file saxpy)
31778: trying file=/run/opengl-driver/lib/libc.so.6
31778: trying file=/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/libc.so.6
31778: trying file=/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/libc.so.6
31778: search path=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib (system search path)
31778: trying file=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6
31778:
31778:
31778: calling init: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/ld-linux-x86-64.so.2
31778:
31778:
31778: calling init: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6
31778:
31778:
31778: calling init: /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libgcc_s.so.1
31778:
31778:
31778: calling init: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libm.so.6
31778:
31778:
31778: calling init: /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
31778:
31778:
31778: calling init: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libpthread.so.0
31778:
31778:
31778: calling init: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/librt.so.1
31778:
31778:
31778: calling init: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libdl.so.2
31778:
31778:
31778: calling init: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcublasLt.so.12
31778:
31778: find library=libcuda.so.1 [0]; searching
31778: search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib (LD_LIBRARY_PATH)
31778: trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcuda.so.1
31778:
31778:
31778: calling init: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcuda.so.1
31778:
31778:
31778: calling init: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcudart.so.12
31778:
31778:
31778: calling init: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcublas.so.12
31778:
31778:
31778: initialize program: saxpy
31778:
31778:
31778: transferring control: saxpy
31778:
Start
Runtime version: 12020
Driver version: 0
Host memory initialized, copying to the device
CUDA error at cudaMalloc(&xDevice, N * sizeof(float)): CUDA driver is a stub library
31778:
31778: calling fini: [0]
31778:
31778:
31778: calling fini: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcublas.so.12 [0]
31778:
31778:
31778: calling fini: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcudart.so.12 [0]
31778:
31778:
31778: calling fini: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcublasLt.so.12 [0]
31778:
31778:
31778: calling fini: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libdl.so.2 [0]
31778:
31778:
31778: calling fini: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/librt.so.1 [0]
31778:
31778:
31778: calling fini: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libpthread.so.0 [0]
31778:
31778:
31778: calling fini: /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6 [0]
31778:
31778:
31778: calling fini: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libm.so.6 [0]
31778:
31778:
31778: calling fini: /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libgcc_s.so.1 [0]
31778:
31778:
31778: calling fini: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6 [0]
31778:
31778:
31778: calling fini: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/ld-linux-x86-64.so.2 [0]
31778:
31778:
31778: calling fini: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcuda.so.1 [0]
31778:
| 15:14:18 |
| Titus | i'll keep updating the gist with the output of the commands that you make me run for anyone joining the party and trying to get a complete picture :)
https://gist.github.com/Titus-von-Koeller/e541c1175b0a191bac75b72d9964e9d0 | 15:29:54 |
 hexa | https://github.com/NixOS/nixpkgs/pull/323154#event-13334193852 | 16:57:30 |
| 29 Jun 2024 |
| SomeoneSerge (matrix works sometimes) | In reply to @titus-von-koeller:matrix.org
i'll keep updating the gist with the output of the commands that you make me run for anyone joining the party and trying to get a complete picture :)
https://gist.github.com/Titus-von-Koeller/e541c1175b0a191bac75b72d9964e9d0
31778:
31778: find library=libcuda.so.1 [0]; searching
31778: search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib (LD_LIBRARY_PATH)
| 02:19:10 |
| SomeoneSerge (matrix works sometimes) | You've set an LD_LIBRARY_PATH to cudatoolkit which currently, after a certain regression, inlcudes the stub driver | 02:19:46 |
| SomeoneSerge (matrix works sometimes) | LD_LIBRARY_PATH is often a footgun | 02:20:05 |
| SomeoneSerge (matrix works sometimes) | * LD_LIBRARY_PATH is often a footgun, it has a higher priority than DT_RUNPATH (which would have lead the linker to /run/opengl-dirver/lib/libcuda.so | 02:20:29 |
| SomeoneSerge (matrix works sometimes) | Unset LD_LIBRARY_PATH or remove cudatoolkit from there | 02:20:38 |
 connor (burnt/out) (UTC-8) | Hey all; I expect I'll be away for about a week starting in a few hours as I try to find housing on the other side of the country :) | 07:36:26 |
|  @nscnt:matrix.org left the room. | 08:09:04 |
