| 3 Jul 2024 |
 hexa | if there was a cache behind nix-community hydra, than you'd be mirroring cache.nixos.org effecitvely | 19:13:45 |
| hexa | * if there was a cache behind nix-community hydra, than you'd be mirroring cache.nixos.org effectively | 19:13:47 |
 SomeoneSerge (matrix works sometimes) | Yeah... Ideally we'd have a solution that evaluates the full DAGs for vanilla and cuda nixpkgs, starts building cuda from the leaves (ehhh, the roots), and always suspends the build if it hash matches the vanilla hash | 19:23:45 |
 connor (burnt/out) (UTC-8) | Is there an open collective for the community hydra instance | 19:24:40 |
| connor (burnt/out) (UTC-8) | If they’re going to be building CUDA packages I definitely want to contribute lol | 19:25:56 |
 matthewcroughan | How can onnxruntime use more than 64G of memory with 12 cores | 22:28:09 |
| matthewcroughan | ugh | 22:28:09 |
| matthewcroughan | oh, maybe because I lack https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/development/libraries/onnxruntime/default.nix#L191 | 22:30:58 |
| matthewcroughan | I think it reaches a part of the build where it runs away | 22:31:15 |
| matthewcroughan | * I think it reaches a part of the build where it runs away, without this | 22:31:20 |
| matthewcroughan | https://github.com/NixOS/nixpkgs/pull/304069 | 22:32:05 |
| matthewcroughan | this | 22:32:06 |
| 4 Jul 2024 |
 Jonas Chevalier | In reply to @connorbaker:matrix.org
Is there an open collective for the community hydra instance yes, we spend it all on hardware: https://opencollective.com/nix-community
we could also explore hardware donation if you want to bring esoteric hardware to the build farm.
| 08:06:20 |
| Jonas Chevalier | In reply to @ss:someonex.net
H'm so in hydra you "create a jobset" somewhere like in a web ui before you merge the terraform configs? Or the tf config is the whole thing but you deployed it manually? The jobset is created with Terraform with https://github.com/nix-community/infra/blob/master/terraform/hydra-projects.tf
This works well because Hydra is a mix of stateful stuff so having a convergence engine is quite nice there.
| 08:07:37 |
| Jonas Chevalier | In reply to @ss:someonex.net
Jonas Chevalier while at it, nobody is building import <nixpkgs> { config.rocmSupport = true; } either, and that one is free Ok, let's do that once CUDA is stable. Building unfreeRedistributable could also be nice. | 08:08:18 |
| Jonas Chevalier | In reply to @ss:someonex.net
Why not just https://github.com/NixOS/nixpkgs/pull/324379/files#diff-b3a88f86f137f8870849673fb9b06582cb73937114ee34a61ae5604e259829a5R37 I think this is going to break our instance. The main hydra needs 128GB of RAM to evaluate all of nixpkgs. If you want to keep the list up to date, it's probably better to invest in a script. | 08:11:25 |
| Jonas Chevalier | In reply to @ss:someonex.net
Why not just https://github.com/NixOS/nixpkgs/pull/324379/files#diff-b3a88f86f137f8870849673fb9b06582cb73937114ee34a61ae5604e259829a5R37 * I think this is going to break our instance. The main hydra needs 128GB of RAM to evaluate all of nixpkgs. If you want to keep the list up to date, it's probably better to invest in a script (that you run locally and commit the result). | 08:11:45 |
|  Philip Taron (UTC-8) left the room. | 15:46:31 |
| 5 Jul 2024 |
| Jonas Chevalier | I don't know if this has been discussed before: did you look at aligning the package versions with some upstream?
For example Nvidia are releasing the nvcr.io Docker images. If we could provide the same versions as a package sets, it would reduce the switching cost for those users. | 06:10:17 |
| SomeoneSerge (matrix works sometimes) | In reply to @zimbatm:numtide.com
I don't know if this has been discussed before: did you look at aligning the package versions with some upstream?
For example Nvidia are releasing the nvcr.io Docker images. If we could provide the same versions as a package sets, it would reduce the switching cost for those users. Well if we're talking about cudaPackages, they are aligned with the manifests that upstream advertises | 07:20:34 |
| SomeoneSerge (matrix works sometimes) |
it would reduce the switching cost for those users.
Do you have a specific example in mind? | 07:20:53 |
| SomeoneSerge (matrix works sometimes) | *
it would reduce the switching cost for those users.
Do you have a specific user story in mind? | 07:21:02 |
| Jonas Chevalier | A very specific example is: one customer is using nvcr.io/nvidia/pytorch:23.08-py3 ( CUDA 12.21, cuDNN 8.9.4, Python 3.10, PyTorch 2.1.0 ) and looking to try out Nix to fix their reproducibility issues | 07:29:17 |
| SomeoneSerge (matrix works sometimes) | And in this case you'd suggest we provide cudaPackages'.cuda_12_21_cudnn_8_9_4? | 07:44:11 |
| SomeoneSerge (matrix works sometimes) | ...instead of referring to the manual and cudaPackages.overrideScope' (...)? | 07:44:49 |
| SomeoneSerge (matrix works sometimes) | * ...instead of referring to the manual and cudaPackages.overrideScope' (...) | 07:44:51 |
| Jonas Chevalier | I haven't thought about this deeply. One potentiality is to maintain a packageset like cudaPackages.pytorch_23_08 | 07:57:44 |
| SomeoneSerge (matrix works sometimes) | I think an out-of-tree collection of buildLayeredImage expressions reproducing nvcr images would make sense | 08:08:38 |
| SomeoneSerge (matrix works sometimes) | In-tree, maybe not so much because these sound like finalized compositions of packages | 08:09:10 |
| hexa | and also Python 3.10 might be hit or miss these days | 10:52:20 |
