| 31 Oct 2024 |
|  Frederik Semmel joined the room. | 15:16:14 |
| Frederik Semmel | In reply to @connorbaker:matrix.org
Absolute biggest pain in my ass right now is packaging onnxruntime (https://github.com/ConnorBaker/cuda-packages/blob/main/cudaPackages-common/onnxruntime/package.nix).
For the ONNX and TensorRT ecosystem, I’m doing a cursed build where the CMake and Python builds are interlaced.
Turns out doing a straight CMake build gives different results compared to doing a Python build. Go figure, the multi-thousand line Python scripts invoked by setup.py change how stuff is configured. Hi connor (he/him) (UTC-7) , first off, the Nix CUDA team is doing awesome work! I have spent the last week reading everything cuda related in nixpkgs and watching NixCon talks :) I am also trying to run onnxruntime with the tensorrt backend on a jetson device with NixOS. Really looking forward to how you might solve the puzzle there. I am new to Nix, therefore I hoped it would be simpler for me to patch the onnxruntime-gpu binaries provided by jetson zoo. My approach is:
- use poetry2nix to pull the specific wheel and run patchelf
- use an poetry2nix overlay to add the cuda packages from jetpack-nixos to the
buildInputs of onnxruntime-gpu
Is this doable like this, do you recommend this? I find it amazing that torch from nixpkgs just works thanks to cuda_compat. Can the cuda_compat trick work here too? Thanks a lot for your time!
| 15:44:26 |
|  didiercrunch joined the room. | 15:57:40 |
 connor (burnt/out) (UTC-8) | I have no idea if it’ll work, but please do try and let me know! I’m not familiar with the Jetson zoo, or how the binary wheels are packaged.
It sounds reasonable — the wheel should have libraries built to target Jetson devices which link against CUDA libraries, and using autoPatchelfHook and including CUDA libraries in buildInputs should patch the libraries so they resolve to the ones provided by Nixpkgs…
I don’t fully understand what cuda_compat does, but my understanding is that it serves as a shim between newer CUDA libraries and an older CUDA driver on the Jetson?
At any rate, try it out and let me know! | 16:16:51 |
| Frederik Semmel | Ok awesome, thanks for the quick reply! At least there is not something completely wrong with the approach. I will let you know if I can make it work 👌 Did anyone else get onnxruntime working with the tensorrt execution provider on a jetson? | 16:24:25 |
| connor (burnt/out) (UTC-8) | I can’t remember if I managed to get it building on Jetson, I got distracted and started doing work on the nix interpreter | 16:26:15 |
| connor (burnt/out) (UTC-8) | Also, which Jetson generation are you using? Please let it be at least Xavier :( | 16:27:48 |
| Frederik Semmel | Its the Orin NX luckily :) I am free to choose a different platform, I am still evaluating what is best for our AI on the edge usecase. From what you say it seems like its not going to be impossible, so I will work my way through it. I hope I can find a a good solution, if I do I will try to contribute it to nixpkgs or jetpack-nixos | 16:31:01 |
 search-sense | I am sorry ``nix run github:SomeoneSerge/pkgs#pkgsCuda.some-pkgs-py.stable-diffusion-webui``` doesn't work anymore | 17:06:45 |
| search-sense | * I am sorry nix run github:SomeoneSerge/pkgs#pkgsCuda.some-pkgs-py.stable-diffusion-webui doesn't work anymore | 17:07:06 |
 SomeoneSerge (back on matrix) | In reply to @search-sense:matrix.org
I am sorry nix run github:SomeoneSerge/pkgs#pkgsCuda.some-pkgs-py.stable-diffusion-webui doesn't work anymore Yeah didn't keep track of it, must've broke with a nixpkgs bump | 17:13:06 |
| search-sense | I unfourtanetly lost the working commit for this repo | 17:25:04 |
| connor (burnt/out) (UTC-8) | In reply to @ss:someonex.net
Hey wanna publish a review on exposing extendMkDerivationArgs? Was this in reference to an existing PR or something new?
I apologize, I've been really scattered the last two weeks :l | 17:57:28 |
| SomeoneSerge (back on matrix) | In reply to @connorbaker:matrix.org
Was this in reference to an existing PR or something new?
I apologize, I've been really scattered the last two weeks :l https://github.com/NixOS/nixpkgs/pull/350350 | 19:38:48 |
| SomeoneSerge (back on matrix) | Huh, seems like there was a simultaneous chagne | 19:39:37 |
| SomeoneSerge (back on matrix) | * Huh, seems like there was a simultaneous change | 19:39:39 |
 Gaétan Lepage | If you happen to just wake of from 6 months spent in coma, don't worry, the triton repo still hasn't got any more tags. | 22:44:38 |
| Gaétan Lepage | 
Download clipboard.png | 22:44:41 |
| Gaétan Lepage | 7 months | 22:45:05 |
| Gaétan Lepage | * 7 months* | 22:45:07 |
| Gaétan Lepage | * If you happen to just wake of from 6 months spent in coma, don't worry, the triton repo still hasn't got any new tag. | 22:45:22 |
| 1 Nov 2024 |
| SomeoneSerge (back on matrix) | I don't think anything short of a github action publishing automatic tags in release branches is going to work with this organization:) | 07:40:32 |
| SomeoneSerge (back on matrix) | That might not either but stands a chance (they merged stuff before) | 07:40:56 |
|  Guillaume Desforges changed their profile picture. | 21:19:14 |
| 2 Nov 2024 |
 sielicki | I do not really follow the conversation on https://github.com/NixOS/nixpkgs/pull/343706 | 23:22:57 |
| sielicki | I don't think it actually matters in this case but just want to call it out if it ever comes up again in the context of nixpkgs: to support the cuda stubs (where you can successfully link a binary against the cuda driver api with the cuda toolkit alone, despite that it doesn't include the full UMD blob), on normal distributions nvidia creates an /usr/local/cuda/lib64/stubs/libcuda.so files, which claim a SONAME with .1, but where the file on disk doesn't actually have that. That means the dynamic loader will never accidentally confuse the stub with the real library. | 23:23:59 |
| SomeoneSerge (back on matrix) | Huh | 23:25:10 |
| sielicki | but now, for some of the libraries that got relinked here, I think now there's two files that have the same name, and im not sure what the dynamic loader does | 23:25:29 |
| sielicki | * but now, for some of the libraries that got relinked here, I think now there's two files that have the same name, and im not sure what the dynamic loader does
edit: maybe not under nixos but potentially for people doing nix on ubuntu or whatever | 23:25:52 |
| sielicki | here let me find you a link, one sec | 23:26:10 |
