| 31 Oct 2024 |
|  Frederik Semmel joined the room. | 15:16:14 |
| Frederik Semmel | In reply to @connorbaker:matrix.org
Absolute biggest pain in my ass right now is packaging onnxruntime (https://github.com/ConnorBaker/cuda-packages/blob/main/cudaPackages-common/onnxruntime/package.nix).
For the ONNX and TensorRT ecosystem, I’m doing a cursed build where the CMake and Python builds are interlaced.
Turns out doing a straight CMake build gives different results compared to doing a Python build. Go figure, the multi-thousand line Python scripts invoked by setup.py change how stuff is configured. Hi connor (he/him) (UTC-7) , first off, the Nix CUDA team is doing awesome work! I have spent the last week reading everything cuda related in nixpkgs and watching NixCon talks :) I am also trying to run onnxruntime with the tensorrt backend on a jetson device with NixOS. Really looking forward to how you might solve the puzzle there. I am new to Nix, therefore I hoped it would be simpler for me to patch the onnxruntime-gpu binaries provided by jetson zoo. My approach is:
- use poetry2nix to pull the specific wheel and run patchelf
- use an poetry2nix overlay to add the cuda packages from jetpack-nixos to the
buildInputs of onnxruntime-gpu
Is this doable like this, do you recommend this? I find it amazing that torch from nixpkgs just works thanks to cuda_compat. Can the cuda_compat trick work here too? Thanks a lot for your time!
| 15:44:26 |
|  didiercrunch joined the room. | 15:57:40 |
 connor (he/him) | I have no idea if it’ll work, but please do try and let me know! I’m not familiar with the Jetson zoo, or how the binary wheels are packaged.
It sounds reasonable — the wheel should have libraries built to target Jetson devices which link against CUDA libraries, and using autoPatchelfHook and including CUDA libraries in buildInputs should patch the libraries so they resolve to the ones provided by Nixpkgs…
I don’t fully understand what cuda_compat does, but my understanding is that it serves as a shim between newer CUDA libraries and an older CUDA driver on the Jetson?
At any rate, try it out and let me know! | 16:16:51 |
| Frederik Semmel | Ok awesome, thanks for the quick reply! At least there is not something completely wrong with the approach. I will let you know if I can make it work 👌 Did anyone else get onnxruntime working with the tensorrt execution provider on a jetson? | 16:24:25 |
| connor (he/him) | I can’t remember if I managed to get it building on Jetson, I got distracted and started doing work on the nix interpreter | 16:26:15 |
| connor (he/him) | Also, which Jetson generation are you using? Please let it be at least Xavier :( | 16:27:48 |
| Frederik Semmel | Its the Orin NX luckily :) I am free to choose a different platform, I am still evaluating what is best for our AI on the edge usecase. From what you say it seems like its not going to be impossible, so I will work my way through it. I hope I can find a a good solution, if I do I will try to contribute it to nixpkgs or jetpack-nixos | 16:31:01 |
 search-sense | I am sorry ``nix run github:SomeoneSerge/pkgs#pkgsCuda.some-pkgs-py.stable-diffusion-webui``` doesn't work anymore | 17:06:45 |
