I was thinking that we might be able to improve the situation by making general nixpkgs contributors more aware of this situation. For example, it would be pretty cool if we could track the nix-community hydra builds on status.nixos.org and on zh.fail (and try to include CUDA packages in future ZHF events).

You're certainly right, and the idea of promoting cuda fixes during ZHF has in fact been around. By the same token, an ofborg-like integration, an external service that would test a PR on-push and post a report on failures on non-default instantiations or involving out-of-tree tests is maybe even necessary to ensure stability of hw-accelerated packages. Even when a contributor doesn't care about cuda, it's important they are informed about unintended consequences of their changes, and maybe can ping the interested parties as needed

For example, https://hydra.nix-community.org/jobset/nixpkgs/cuda/evals has a bunch of Eval Errors and build errors and I don't remember the last time that it was green (although some of those eval errors might not be indicative of actually broken packages).

My javascript might be broken, but I only see build failures. Some errors under cudaPackages. seem actually familiar, e.g. the cutensor error was fixed at least once already and is recurring... that's to be fixed somewhere around manifest.nix in the current implementation

Also, I understand why hydra.nixos.org doesn't build CUDA packages, but do you think that we could enable evaluation-only checks for CUDA packages on nixpkgs github PRs and then build those PRs using the nix-community builders and report the results on the PR?

Ah great, you already said as much. Yes, we definitely can. You may have seen issues about unfree stuff open and closed in the Ofborg repo, so the notion isn't entirely new. I know for sure there are several interested parties, and this would be incredibly useful, maybe we can discuss in more detail on the call. This issue needs to be approached with some from the community perspective though, because it's desirable for nixpkgs and nix-community to still stay independent/disentangled: legally, socially, architecturally...

       > ERROR: noBrokenSymlinks: the symlink /nix/store/fqx2dv9vp1k0f00imgqshy6d92ykcw5d-python3.12-kaleido-0.2.1/lib/python3.12/site-packages/kaleido/executable/etc/fonts/fonts.conf points to a missing target /nix/store/2ynwbywyaxk4wgl8d3xrb9dzkdzv241x-fontconfig-2.15.0-bin/etc/fonts/fonts.conf
       > ERROR: noBrokenSymlinks: found 1 dangling symlinks and 0 reflexive symlinks
       For full logs, run 'nix log /nix/store/f7whd4p85k8b7bd8sx2bnp5jpmzycbkx-python3.12-kaleido-0.2.1.drv'.
error: 1 dependencies of derivation '/nix/store/gdg8kgy8ry0gjhpv2dws072wajkjk69l-python3.12-plotly-5.24.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/pfxw0g0npwr091cr7ks7012jl8qsg

Regarding the remaining 21 eval errors:

• 13x cuda-samples depends on freeimage which is technically insecure. I am not 100% sure, if we should just filter all of the cuda-samples packages using the new filterPackagePredicates mechanism or if it might be better to do

  freeimage.overrideAttrs { meta.knownVulnerabilities = [ ]; }
  

  since cuda-samples isn't really "production-facing" anyway, so the users should only really care that the samples compile. The security risk of distributing sample code that technically has some vulnerabilities should be minimal.

• colmap also depends on freeimage, this issue should be probably raised upstream

• boxx and bpycv haven't been updated upstream in the last 11 months and they seem to not support any of the python versions that are currently supported in nixpkgs. So we should probably check in with the nixpkgs maintainer and remove these packages if they aren't required by something important.

• pixinsight is (and always was) unfree, but it is explicitly listed in release-cuda.nix for some reason. Should it be removed?

• tts because it depends on a -bin version of pytorch for some reason, which is "unfree" (bsd3 issl unfreeRedistributable). Is it possible to make it depend on a non-binary version of pytorch or should it be removed from release-cuda.nix?

• mxnet is "actually" broken since #173463

• truecrack-cuda is "actually" broken since #167250

• pymc depends on pytensor is "actually" broken since #373239

Regarding the remaining 21 eval errors:

• 13x cuda-samples depends on freeimage which is technically insecure. I am not 100% sure, if we should just filter all of the cuda-samples packages using the new filterPackagePredicates mechanism or if it might be better to do

freeimage.overrideAttrs { meta.knownVulnerabilities = [ ]; }

specifically for cuda-samples. Since cuda-samples isn't really "production-facing" anyway, so the users should only really care that the samples compile. The security risk of distributing sample code that technically has some vulnerabilities should be minimal.

• colmap also depends on freeimage, this issue should be probably raised upstream
• boxx and bpycv haven't been updated upstream in the last 11 months and they seem to not support any of the python versions that are currently supported in nixpkgs. So we should probably check in with the nixpkgs maintainer and remove these packages if they aren't required by something important.
• pixinsight is (and always was) unfree, but it is explicitly listed in release-cuda.nix for some reason. Should it be removed?
• tts because it depends on a -bin version of pytorch for some reason, which is "unfree" (bsd3 issl unfreeRedistributable). Is it possible to make it depend on a non-binary version of pytorch or should it be removed from release-cuda.nix?
• mxnet is "actually" broken since #173463
• truecrack-cuda is "actually" broken since #167250
• pymc depends on pytensor is "actually" broken since #373239