| 4 Mar 2025 |
 stick | an ideal thing for me would be if someone indicated the list of packages that really need to have the build fixed before the merge happens and I would (try to) work on fixing these | 10:53:23 |
| stick | * an ideal thing for me would be if someone indicated the list of packages that really need to have the build fixed before the merge happens and I will (try to) work on fixing these | 10:53:33 |
 SomeoneSerge (back on matrix) | In addition to Connor's suggestions, can you check what is the output when you run cudaPackages.saxpy? | 11:26:55 |
| stick | Maybe the merge of this PR should happen shortly after the merge of ROCm update in #367695 to not do massive rebuilds two times? | 12:12:00 |
| stick | * Maybe the merge of this PR should happen shortly after the merge of ROCm update in #367695 to not do massive rebuilds two times?
https://github.com/NixOS/nixpkgs/pull/367695 | 12:12:17 |
| 5 Mar 2025 |
|  asa set a profile picture. | 08:07:57 |
| asa changed their display name from Asa to asa. | 08:08:10 |
| 7 Mar 2025 |
|  mdietrich joined the room. | 13:03:38 |
| mdietrich | Hey all, first of all thank you for your work, last time I tried to use any cuda-related programs and services I had to give up because this joint effort had not been set up.
I am just wondering if I am doing something wrong when trying to set up llama-cpp and open-webui on my NixOS machine. I've set up the nix-community cache (and ollama with CUDA support installs fine in any case), but either enabling nixpkgs.config.cudaSupport or overwriting e.g. llama-cpp's package with `services.llama-cpp.package = pkgs.overwrite { config.cudaSupport = true; config.rocmSupport = false; } | 13:12:26 |
| mdietrich | * Hey all, first of all thank you for your work, last time I tried to use any cuda-related programs and services I had to give up because this joint effort had not been set up.
I am just wondering if I am doing something wrong when trying to set up llama-cpp and open-webui on my NixOS machine. I've set up the nix-community cache (and ollama with CUDA support installs fine in any case), but neither enabling nixpkgs.config.cudaSupport or overwriting e.g. llama-cpp's package with `services.llama-cpp.package = pkgs.overwrite { config.cudaSupport = true; config.rocmSupport = false; }` just dowload and install the appropriate packages, but lead to extremely long build times. Are these packages (llama-cpp and open-webui, of which I think onnxruntime takes the longest) just not built in the community cache? | 13:13:54 |
| SomeoneSerge (back on matrix) | Let's see | 13:15:53 |
| SomeoneSerge (back on matrix) | https://hydra.nix-community.org/job/nixpkgs/cuda/llama-cpp.x86_64-linux | 13:15:55 |
| SomeoneSerge (back on matrix) | https://hydra.nix-community.org/job/nixpkgs/cuda/onnxruntime.x86_64-linux | 13:16:26 |
| SomeoneSerge (back on matrix) | open-webui apparently wasn't added to the release-cuda.nix file yet: https://hydra.nix-community.org/job/nixpkgs/cuda/open-webui.x86_64-linuxd | 13:17:10 |
| SomeoneSerge (back on matrix) | As for onnxruntime and llama-cpp, let's compare the hashes in your llama-cpp and the one reported by hydra | 13:18:20 |
| mdietrich | I am on x86_64, nixos-unstable with flakes with an RTX 3060 Ti and following substituters:
substituters = [
"https://nix-community.cachix.org"
];
trusted-public-keys = [
"nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs="
];
| 13:19:29 |
| mdietrich | Thank you for your quick answer | 13:20:01 |
| mdietrich | I think onnxruntime is a dependency of open-webui and not llama-cpp, open-webui itself probably (?) does not need cuda support itself | 13:20:45 |
| mdietrich | services.llama-cpp.package has the value "«derivation /nix/store/dhqdwqp6akr6h6f1k3rz190m3syrv6iy-llama-cpp-4731.drv»" | 13:23:06 |
| SomeoneSerge (back on matrix) | Let's try nix path-info --override-input nixpkgs github:NixOS/nixpkgs/1d2fe0135f360c970aee1d57a53f816f3c9bddae --derivation .#nixosConfigurations.$(hostname).config.services.llama-cpp.package to make it comparable with https://hydra.nix-community.org/build/3552955#tabs-buildinputs | 13:24:11 |
| SomeoneSerge (back on matrix) | I'd maybe not focus on these concerns, the expert hours are arguably more expensive that rebuild costs | 13:25:27 |
| SomeoneSerge (back on matrix) | (still pending =) | 13:25:51 |
| mdietrich | Wait a minute, I am slightly confused as llama-cpp seems to actually have cuda support now that I rebuilt a couple of minutes ago. It just does not use my GPU when running inference even though it reports it as visible and usable. Maybe a configuration mistake on my side (although I am using the default NixOS service). I'll look into open-webui and onnxruntime now... | 13:27:31 |
| mdietrich | Yes, onnxruntime does recompile, as well as python3.12-torch-2.5.1. I'm checkin the hashes now... | 13:33:36 |
| mdietrich | I am definitely building onnxruntime myself even though I get:
> nix path-info --override-input nixpkgs github:NixOS/nixpkgs/9f41a78ead0fbe2197cd4c09b5628060456cd6e3 --derivation .\#nixosConfigurations.$(hostname).pkgs.onnxruntime
• Updated input 'nixpkgs':
'github:nixos/nixpkgs/32fb99ba93fea2798be0e997ea331dd78167f814?narHash=sha256-ozoOtE2hGsqh4XkTJFsrTkNxkRgShxpQxDynaPZUGxk%3D' (2025-02-21)
→ 'github:NixOS/nixpkgs/9f41a78ead0fbe2197cd4c09b5628060456cd6e3?narHash=sha256-WWXRCTOWcKvtzqzVgBMON0/TWcFMyWq831HQUITE4rs%3D' (2025-02-21)
/nix/store/a22vqi9d0ndhlcy1yxw4m3ir4z7ckfrg-onnxruntime-1.20.1.drv
Which is the same hash as the hydra build store path
| 13:48:48 |
| mdietrich | I get the same hash for pytorch locally and in hydra as well! | 13:56:11 |
| SomeoneSerge (back on matrix) | And if you nix build --override-input nixpkgs github:NixOS/nixpkgs/9f41a78ead0fbe2197cd4c09b5628060456cd6e3 .#nixosConfigurations.$(hostname).pkgs.onnxruntime? | 13:59:42 |
| mdietrich | Then I'm building nccl and cudnn-frontend for some reason? | 14:15:15 |
| SomeoneSerge (back on matrix) | Well this certainly shouldn't be happening if the hashes indeed match | 14:21:40 |
| SomeoneSerge (back on matrix) | Which hydra eval did you refer to? | 14:22:00 |
