| 25 Jul 2024 |
 ˈt͡sɛːzaɐ̯ | I can't reproduce that on a nixos, github:NixOS/nixpkgs/289eafaaf02177e5814d8738cf57e259f9eae46e#ollama-cuda builds fine. (should master from an ~hour ago.) | 06:26:48 |
| ˈt͡sɛːzaɐ̯ | * I can't reproduce that on a nixos, github:NixOS/nixpkgs/289eafaaf02177e5814d8738cf57e259f9eae46e#ollama-cuda builds fine. (should master from half an ~hour ago.) | 06:27:09 |
| ˈt͡sɛːzaɐ̯ | * Pascal: I can't reproduce that on a nixos, github:NixOS/nixpkgs/289eafaaf02177e5814d8738cf57e259f9eae46e#ollama-cuda builds fine. (should master from half an ~hour ago.) | 06:27:22 |
 Pascal | In reply to @julius:mtx.liftm.de
Pascal: I can't reproduce that on a nixos, github:NixOS/nixpkgs/289eafaaf02177e5814d8738cf57e259f9eae46e#ollama-cuda builds fine. (should master from half an ~hour ago.) Thanks! I'll give that one a go later. | 09:17:32 |
| Pascal | @julius:mtx.liftm.de: Just checked: Works on NixOS, but still fails on Ubuntu (using nix run --impure). Same error: crt/host_config.h not found. Dang... it worked until 3 weeks or so ago.
It fails during 'Compiling the CUDA compiler identification source file' oddly using /usr/local/cuda/bin/nvcc despite finding /nix/store/...-cuda_nvcc... 🤷♂️ | 15:07:36 |
 SomeoneSerge (utc+3) | In reply to @phirsch:matrix.org
@julius:mtx.liftm.de: Just checked: Works on NixOS, but still fails on Ubuntu (using nix run --impure). Same error: crt/host_config.h not found. Dang... it worked until 3 weeks or so ago.
It fails during 'Compiling the CUDA compiler identification source file' oddly using /usr/local/cuda/bin/nvcc despite finding /nix/store/...-cuda_nvcc... 🤷♂️ Try entering a shell with cudaPackages.cuda_nvcc I guess | 15:13:03 |
| Pascal | Since I don't have any CUDA env vars set nor nvcc on the PATH, I suspect that the nvcc location is explicitly specified in the derivation's build script... | 15:13:14 |
| SomeoneSerge (utc+3) | I didn't check but I'd guess ollama doesn't propagate nvcc (not necessarily bad) and then it ends up picking up the bad nvcc from ubuntu (/usr/...) | 15:14:17 |
| Pascal | Now this is weird: Simply running
nix run github:NixOS/nixpkgs/289eafaaf02177e5814d8738cf57e259f9eae46e#ollama serve
(without -cuda!) works, showing 'library=cuda' and accurate GPU info... Can't claim to understand why, but it looks like I don't actually need the '-cuda' variant. (Suspect that it's somehow managing to dload() the system CUDA SOs?) | 15:31:57 |
| Pascal | @ss:someonex.net: Thanks for the tips (only saw your replies after refreshing just now). Might try that as well (although I might already be good according to the above). | 15:41:55 |
| 26 Jul 2024 |
|  Jason Schnitzer joined the room. | 01:35:00 |
|  nathan72419 joined the room. | 16:11:45 |
 connor (he/him) (UTC-7) | As a heads up, I’m moving this week so my availability will probably be more limited than it was already for a few weeks :/ | 16:27:09 |
| 27 Jul 2024 |
| Pascal | @SomeoneSerge (UTC+3) @ˈt͡sɛːzaɐ̯ No dice... While ollama (without '-cuda') somehow manages to get GPU serial and VRAM allocation into, it doesn't use the GPU when actually running a model (outputs 'Not compiled with GPU offload support'). And unfortunately, using 'nix run --impure' as above from within a nix shell with 'nvcc' from nixpkgs still fails because it's using nvcc from /usr/local/...
| 07:04:48 |
| 28 Jul 2024 |
|  matthewcroughan changed their display name from matthewcroughan to matthewcroughan - going to nix.camp. | 16:12:17 |
| 29 Jul 2024 |
|  kaya changed their profile picture. | 07:47:57 |
|  @ironbound:hackerspace.pl removed their display name P_Big. | 14:47:13 |
| @ironbound:hackerspace.pl removed their profile picture. | 14:47:26 |
| 30 Jul 2024 |
 srhb | I'm waving around a big hammer here. Does anyone want to save cuda-modules/aliases.nix? 😁
https://github.com/NixOS/nixpkgs/pull/331017 | 05:57:45 |
| SomeoneSerge (utc+3) | In reply to @srhb:matrix.org
I'm waving around a big hammer here. Does anyone want to save cuda-modules/aliases.nix? 😁
https://github.com/NixOS/nixpkgs/pull/331017 Yes we want to save aliases. I think the solution should be to ensure these attributes are kept lazy, not to remove them | 07:06:42 |
| srhb | Sad. Alright, I'll draft them :) | 07:06:57 |
| srhb | The aliases I nuked are still OK to go by now, right? | 07:07:11 |
| SomeoneSerge (utc+3) | I need to fetch my laptop:) | 07:07:36 |
| srhb | And laziness won't save that torch check, right? (equality has no choice but to strict) | 07:07:54 |
| srhb | Though I suppose I could exempt those exact attributes in the torch check. Lots of spooky action at a distance though. | 07:09:22 |
| SomeoneSerge (utc+3) | Ooh, that, the package set comparison. I forgot it was there | 07:10:30 |
| srhb | I understand why it's there, but I think it should go. | 07:11:48 |
| SomeoneSerge (utc+3) | Yea the check is quite a bit of a heuristic actually | 07:12:14 |
| SomeoneSerge (utc+3) | In reply to @srhb:matrix.org
The aliases I nuked are still OK to go by now, right? Yes agreed | 07:12:48 |
| srhb | So my preferred choice of action would be to a) nuke the old aliases, b) keep the alias infrastructure, and ideally c) remove that torch check, because any aliasing will just reintroduce this problem across all tooling that touches torch, producing warnings that may be completely irrelevant as they are in this case. | 07:13:59 |
