| 3 Mar 2025 |
 connor (burnt/out) (UTC-8) | little_dude: it's fine to cross-post!
Sorry it's not working, my only suggestion would be to try running it with whatever flags Ollama needs to enable debugging and/or LD_DEBUG=libs to make sure it's finding and loading the correct libraries.
The version difference across CUDA driver version and CUDA library version is fine -- just means you can run CUDA libraries using up to and including 12.8.
The GPU definitely supports multiple workloads, so that shouldn't be a problem either.
I'm strapped for time so I probably won't be able to help debug or troubleshoot, but I think some other people in here use ollama, so they might be able to chime in.
| 16:46:44 |
| 4 Mar 2025 |
 stick | i have prepared a cudaPackages_12 update from 12.4 to 12.8 here:
https://github.com/NixOS/nixpkgs/pull/386983
can you have a look? I also included a nixpkgs-review result - 229 marked as broken / 219 failed to build / 2455 packages built
but I am having hard time figuring out which build failures are new and which were happening even before
can you advise what is the best way how to proceed? please comment on github, i am not always following the discussion here
| 10:48:13 |
| stick | an ideal thing for me would be if someone indicated the list of packages that really need to have the build fixed before the merge happens and I would (try to) work on fixing these | 10:53:23 |
| stick | * an ideal thing for me would be if someone indicated the list of packages that really need to have the build fixed before the merge happens and I will (try to) work on fixing these | 10:53:33 |
 SomeoneSerge (back on matrix) | In addition to Connor's suggestions, can you check what is the output when you run cudaPackages.saxpy? | 11:26:55 |
| stick | Maybe the merge of this PR should happen shortly after the merge of ROCm update in #367695 to not do massive rebuilds two times? | 12:12:00 |
| stick | * Maybe the merge of this PR should happen shortly after the merge of ROCm update in #367695 to not do massive rebuilds two times?
https://github.com/NixOS/nixpkgs/pull/367695 | 12:12:17 |
| 5 Mar 2025 |
|  asa set a profile picture. | 08:07:57 |
| asa changed their display name from Asa to asa. | 08:08:10 |
| 7 Mar 2025 |
|  mdietrich joined the room. | 13:03:38 |
| mdietrich | Hey all, first of all thank you for your work, last time I tried to use any cuda-related programs and services I had to give up because this joint effort had not been set up.
I am just wondering if I am doing something wrong when trying to set up llama-cpp and open-webui on my NixOS machine. I've set up the nix-community cache (and ollama with CUDA support installs fine in any case), but either enabling nixpkgs.config.cudaSupport or overwriting e.g. llama-cpp's package with `services.llama-cpp.package = pkgs.overwrite { config.cudaSupport = true; config.rocmSupport = false; } | 13:12:26 |
| mdietrich | * Hey all, first of all thank you for your work, last time I tried to use any cuda-related programs and services I had to give up because this joint effort had not been set up.
I am just wondering if I am doing something wrong when trying to set up llama-cpp and open-webui on my NixOS machine. I've set up the nix-community cache (and ollama with CUDA support installs fine in any case), but neither enabling nixpkgs.config.cudaSupport or overwriting e.g. llama-cpp's package with `services.llama-cpp.package = pkgs.overwrite { config.cudaSupport = true; config.rocmSupport = false; }` just dowload and install the appropriate packages, but lead to extremely long build times. Are these packages (llama-cpp and open-webui, of which I think onnxruntime takes the longest) just not built in the community cache? | 13:13:54 |
| SomeoneSerge (back on matrix) | Let's see | 13:15:53 |
| SomeoneSerge (back on matrix) | https://hydra.nix-community.org/job/nixpkgs/cuda/llama-cpp.x86_64-linux | 13:15:55 |
| SomeoneSerge (back on matrix) | https://hydra.nix-community.org/job/nixpkgs/cuda/onnxruntime.x86_64-linux | 13:16:26 |
| SomeoneSerge (back on matrix) | open-webui apparently wasn't added to the release-cuda.nix file yet: https://hydra.nix-community.org/job/nixpkgs/cuda/open-webui.x86_64-linuxd | 13:17:10 |
| SomeoneSerge (back on matrix) | As for onnxruntime and llama-cpp, let's compare the hashes in your llama-cpp and the one reported by hydra | 13:18:20 |
| mdietrich | I am on x86_64, nixos-unstable with flakes with an RTX 3060 Ti and following substituters:
substituters = [
"https://nix-community.cachix.org"
];
trusted-public-keys = [
"nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs="
];
| 13:19:29 |
| mdietrich | Thank you for your quick answer | 13:20:01 |
| mdietrich | I think onnxruntime is a dependency of open-webui and not llama-cpp, open-webui itself probably (?) does not need cuda support itself | 13:20:45 |
| mdietrich | services.llama-cpp.package has the value "«derivation /nix/store/dhqdwqp6akr6h6f1k3rz190m3syrv6iy-llama-cpp-4731.drv»" | 13:23:06 |
| SomeoneSerge (back on matrix) | Let's try nix path-info --override-input nixpkgs github:NixOS/nixpkgs/1d2fe0135f360c970aee1d57a53f816f3c9bddae --derivation .#nixosConfigurations.$(hostname).config.services.llama-cpp.package to make it comparable with https://hydra.nix-community.org/build/3552955#tabs-buildinputs | 13:24:11 |
| SomeoneSerge (back on matrix) | I'd maybe not focus on these concerns, the expert hours are arguably more expensive that rebuild costs | 13:25:27 |
| SomeoneSerge (back on matrix) | (still pending =) | 13:25:51 |
| mdietrich | Wait a minute, I am slightly confused as llama-cpp seems to actually have cuda support now that I rebuilt a couple of minutes ago. It just does not use my GPU when running inference even though it reports it as visible and usable. Maybe a configuration mistake on my side (although I am using the default NixOS service). I'll look into open-webui and onnxruntime now... | 13:27:31 |
| mdietrich | Yes, onnxruntime does recompile, as well as python3.12-torch-2.5.1. I'm checkin the hashes now... | 13:33:36 |
| mdietrich | I am definitely building onnxruntime myself even though I get:
> nix path-info --override-input nixpkgs github:NixOS/nixpkgs/9f41a78ead0fbe2197cd4c09b5628060456cd6e3 --derivation .\#nixosConfigurations.$(hostname).pkgs.onnxruntime
• Updated input 'nixpkgs':
'github:nixos/nixpkgs/32fb99ba93fea2798be0e997ea331dd78167f814?narHash=sha256-ozoOtE2hGsqh4XkTJFsrTkNxkRgShxpQxDynaPZUGxk%3D' (2025-02-21)
→ 'github:NixOS/nixpkgs/9f41a78ead0fbe2197cd4c09b5628060456cd6e3?narHash=sha256-WWXRCTOWcKvtzqzVgBMON0/TWcFMyWq831HQUITE4rs%3D' (2025-02-21)
/nix/store/a22vqi9d0ndhlcy1yxw4m3ir4z7ckfrg-onnxruntime-1.20.1.drv
Which is the same hash as the hydra build store path
| 13:48:48 |
| mdietrich | I get the same hash for pytorch locally and in hydra as well! | 13:56:11 |
| SomeoneSerge (back on matrix) | And if you nix build --override-input nixpkgs github:NixOS/nixpkgs/9f41a78ead0fbe2197cd4c09b5628060456cd6e3 .#nixosConfigurations.$(hostname).pkgs.onnxruntime? | 13:59:42 |
| mdietrich | Then I'm building nccl and cudnn-frontend for some reason? | 14:15:15 |
