For example, https://hydra.nix-community.org/jobset/nixpkgs/cuda/evals has a bunch of Eval Errors and build errors and I don't remember the last time that it was green (although some of those eval errors might not be indicative of actually broken packages).

My javascript might be broken, but I only see build failures. Some errors under cudaPackages. seem actually familiar, e.g. the cutensor error was fixed at least once already and is recurring... that's to be fixed somewhere around manifest.nix in the current implementation

Also, I understand why hydra.nixos.org doesn't build CUDA packages, but do you think that we could enable evaluation-only checks for CUDA packages on nixpkgs github PRs and then build those PRs using the nix-community builders and report the results on the PR?

Ah great, you already said as much. Yes, we definitely can. You may have seen issues about unfree stuff open and closed in the Ofborg repo, so the notion isn't entirely new. I know for sure there are several interested parties, and this would be incredibly useful, maybe we can discuss in more detail on the call. This issue needs to be approached with some from the community perspective though, because it's desirable for nixpkgs and nix-community to still stay independent/disentangled: legally, socially, architecturally...