In reply to @hexa:lossy.network
is nix-community's jobset going to stay a ci essentially?

I'm trying to build the Rust candle-kernels crate within Nix, and nvcc is complaining that gcc is too new. I have gcc 13.2.0 and nvcc version 12.2.140 from nixpkgs-24.05-darwin bb8bdb47b718645b2f198a6cf9dff98d967d0fd4.

  /nix/store/r45hzi56bzljzfvh6rgdnjbisy9pxqnj-cuda-merged-12.2/include/crt/host_config.h:143:2: error: #error -- unsupported GNU version! gcc versions later than 12 are not supported! The nvcc flag '-allow-unsupported-compiler' can be used to override this version check; however, using an unsupported host compiler may cause compilation failure or incorrect run time execution. Use at your own risk.
        |  ^~~~~
  thread 'main' panicked at /home/gmacon3/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen_cuda-0.1.5/src/lib.rs:391:13:
  nvcc error while compiling "src/affine.cu":

  # CLI "nvcc" "--gpu-architecture=sm_90" "--ptx" "--default-stream" "per-thread" "--output-directory" "$PWD/target/debug/build/candle-kernels-809f3e0b9ee8b48d/out" "-Isrc" "-I/nix/store/r45hzi56bzljzfvh6rgdnjbisy9pxqnj-cuda-merged-12.2/include" "src/affine.cu" 

Have other folks seen this? What's the best approach to resolve this?

In reply to @gmacon:matrix.org

I'm trying to build the Rust candle-kernels crate within Nix, and nvcc is complaining that gcc is too new. I have gcc 13.2.0 and nvcc version 12.2.140 from nixpkgs-24.05-darwin bb8bdb47b718645b2f198a6cf9dff98d967d0fd4.

  /nix/store/r45hzi56bzljzfvh6rgdnjbisy9pxqnj-cuda-merged-12.2/include/crt/host_config.h:143:2: error: #error -- unsupported GNU version! gcc versions later than 12 are not supported! The nvcc flag '-allow-unsupported-compiler' can be used to override this version check; however, using an unsupported host compiler may cause compilation failure or incorrect run time execution. Use at your own risk.
        |  ^~~~~
  thread 'main' panicked at /home/gmacon3/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen_cuda-0.1.5/src/lib.rs:391:13:
  nvcc error while compiling "src/affine.cu":

  # CLI "nvcc" "--gpu-architecture=sm_90" "--ptx" "--default-stream" "per-thread" "--output-directory" "$PWD/target/debug/build/candle-kernels-809f3e0b9ee8b48d/out" "-Isrc" "-I/nix/store/r45hzi56bzljzfvh6rgdnjbisy9pxqnj-cuda-merged-12.2/include" "src/affine.cu" 

Have other folks seen this? What's the best approach to resolve this?

older gcc

Note that if you're building a shared library you're going to run into libc issues if you just use gcc12Stdenv. THat's why we have cudaPackages.backendStdenv

In reply to @ss:someonex.net

older gcc

Note that if you're building a shared library you're going to run into libc issues if you just use gcc12Stdenv. THat's why we have cudaPackages.backendStdenv

In reply to @gmacon:matrix.org
Since this is a Rust project, I'm not building any shared libraries, but this is good to know. Thanks!

In reply to @ss:someonex.net
(also relevant if you're loading other shared libraries, e.g. as plugins)

In reply to @zimbatm:numtide.com
hexa (UTC+1): can you give us the new stats after you add that cache :)

hmm, ollama is failing for me on unstable

Sep 07 15:59:47 sink1 ollama[1314]: time=2024-09-07T15:59:47.680-04:00 level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=/srv/fast/ollama/models/blobs/sha256-5ff0abeeac1d2dbdd5455c0b49ba3b29a9ce3c1fb181b2eef2e948689d55d046 gpu=GPU-c2c9209f-9632-bb03-ca95-d903c8664a1a parallel=4 available=12396331008 required="11.1 GiB"
Sep 07 15:59:47 sink1 ollama[1314]: time=2024-09-07T15:59:47.681-04:00 level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=28 layers.offload=28 layers.split="" memory.available="[11.5 GiB]" memory.required.full="11.1 GiB" memory.required.partial="11.1 GiB" memory.required.kv="2.1 GiB" memory.required.allocations="[11.1 GiB]" memory.weights.total="10.1 GiB" memory.weights.repeating="10.0 GiB" memory.weights.nonrepeating="164.1 MiB" memory.graph.full="296.0 MiB" memory.graph.partial="391.4 MiB"
Sep 07 15:59:47 sink1 ollama[1314]: time=2024-09-07T15:59:47.695-04:00 level=INFO source=server.go:391 msg="starting llama server" cmd="/tmp/ollama1289771407/runners/cuda_v12/ollama_llama_server --model /srv/fast/ollama/models/blobs/sha256-5ff0abeeac1d2dbdd5455c0b49ba3b29a9ce3c1fb181b2eef2e948689d55d046 --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 28 --parallel 4 --port 35991"
Sep 07 15:59:47 sink1 ollama[1314]: time=2024-09-07T15:59:47.696-04:00 level=INFO source=sched.go:450 msg="loaded runners" count=1
Sep 07 15:59:47 sink1 ollama[1314]: time=2024-09-07T15:59:47.696-04:00 level=INFO source=server.go:591 msg="waiting for llama runner to start responding"
Sep 07 15:59:47 sink1 ollama[1314]: time=2024-09-07T15:59:47.696-04:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server error"
Sep 07 15:59:47 sink1 ollama[1314]: /tmp/ollama1289771407/runners/cuda_v12/ollama_llama_server: error while loading shared libraries: libcudart.so.12: cannot open shared object file: No such file or directory
Sep 07 15:59:47 sink1 ollama[1314]: time=2024-09-07T15:59:47.947-04:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="llama runner process has terminated: exit status 127"