Let's schedule a call to discuss how to go forward with stdenv support, setup-hooks, wrappers, config.cudaSupport, localSystem/pkgsCross, clang support, and out-of-tree override-ability of manifests and toolkit components https://crab.fit/cudapackages-ng-781527

CC connor (he/him) (UTC-7), sielicki, Samuel Ainsworth, and anyone interested

Starting with the last question: great to hear! As one tool to help with discovery, we have a task board at https://github.com/orgs/NixOS/projects/27/views/1. We haven't been properly maintaining it for the last year, I see many invalidated/outdated items there, but some of the roadmap is still relevant, and the "New" column is automatically populated with all issues and PRs tagged "cuda".

If you're willing to do chores, fixing issues like "nvidia's bash wrapper for nsys-ui assumes things are installed into weird locations and is completely broken" and "a package has changed the way they hardcode /usr/lib or dlopen stuff and now fails to find libcuda.so again", those would be very useful, relatively straightforward, but involve an amount of debugging and suffering and usually get ignored for a long time because it's just demotivating.

If you're interested in architectural issues, then note the message about the upcoming meeting and the proposed subjects, check out the "Roadmap" column, and Connor's out-of-tree cuda-packages

I was thinking that we might be able to improve the situation by making general nixpkgs contributors more aware of this situation. For example, it would be pretty cool if we could track the nix-community hydra builds on status.nixos.org and on zh.fail (and try to include CUDA packages in future ZHF events).

You're certainly right, and the idea of promoting cuda fixes during ZHF has in fact been around. By the same token, an ofborg-like integration, an external service that would test a PR on-push and post a report on failures on non-default instantiations or involving out-of-tree tests is maybe even necessary to ensure stability of hw-accelerated packages. Even when a contributor doesn't care about cuda, it's important they are informed about unintended consequences of their changes, and maybe can ping the interested parties as needed

For example, https://hydra.nix-community.org/jobset/nixpkgs/cuda/evals has a bunch of Eval Errors and build errors and I don't remember the last time that it was green (although some of those eval errors might not be indicative of actually broken packages).

My javascript might be broken, but I only see build failures. Some errors under cudaPackages. seem actually familiar, e.g. the cutensor error was fixed at least once already and is recurring... that's to be fixed somewhere around manifest.nix in the current implementation

Also, I understand why hydra.nixos.org doesn't build CUDA packages, but do you think that we could enable evaluation-only checks for CUDA packages on nixpkgs github PRs and then build those PRs using the nix-community builders and report the results on the PR?

Ah great, you already said as much. Yes, we definitely can. You may have seen issues about unfree stuff open and closed in the Ofborg repo, so the notion isn't entirely new. I know for sure there are several interested parties, and this would be incredibly useful, maybe we can discuss in more detail on the call. This issue needs to be approached with some from the community perspective though, because it's desirable for nixpkgs and nix-community to still stay independent/disentangled: legally, socially, architecturally...

       > ERROR: noBrokenSymlinks: the symlink /nix/store/fqx2dv9vp1k0f00imgqshy6d92ykcw5d-python3.12-kaleido-0.2.1/lib/python3.12/site-packages/kaleido/executable/etc/fonts/fonts.conf points to a missing target /nix/store/2ynwbywyaxk4wgl8d3xrb9dzkdzv241x-fontconfig-2.15.0-bin/etc/fonts/fonts.conf
       > ERROR: noBrokenSymlinks: found 1 dangling symlinks and 0 reflexive symlinks
       For full logs, run 'nix log /nix/store/f7whd4p85k8b7bd8sx2bnp5jpmzycbkx-python3.12-kaleido-0.2.1.drv'.
error: 1 dependencies of derivation '/nix/store/gdg8kgy8ry0gjhpv2dws072wajkjk69l-python3.12-plotly-5.24.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/pfxw0g0npwr091cr7ks7012jl8qsg