In reply to @zowoq:matrix.org
I imagine that the amount and size of builds would make cachix or other cloud storage unfeasible. If it was only a dev cache could probably get away with just serving it off the CI master, if it was a proper public cache with non-trivial amount of users probably want a dedicated machine (or more than one if you want to keep the cache around for a while).

I think we're probably close to users having problems with the cachix cache expiring too quickly. Our setup doesn't allow us to selectively push to the cache, everything that goes through CI gets pushed. Would need to move these jobs to another hydra/machine but that would also avoid needing to deal with this:

ensuring sufficiently smart scheduling to not jam other jobsets

I imagine that the amount and size of builds would make cachix or other cloud storage unfeasible

True. Maybe we should allocate setting up a tvix nar-bridge as a substituter as a separate task, so that public cache can still be a thing xD

Would need to move these jobs to another hydra/machine but that would also avoid needing to deal with this:

Indeed, that's one way to get started at least

Not 100% sure, what do you mean? The problematic cuda versions are 11.0-11.3.

11.4 and later have individual redistributables. 10.x are already deprecated/removed from nixpkgs, so no need to worry about those.

In reply to @ruroruro:matrix.org
Not 100% sure, what do you mean? The problematic cuda versions are 11.0-11.3. 11.4 and later have individual redistributables. 10.x are already deprecated/removed from nixpkgs, so no need to worry about those.

In reply to @ss:someonex.net

Great, thanks! So, the question essentially is: we (I think I say this with the cuda team hat on) can and want to scale up the CI for testing CUDA-enabled packages, both by increasing the number of builders, and by adding GPU instances. We want to build many more variants of nixpkgs for different architectures, and, ideally, run tests across a matrix of co-processor devices. For obvious reasons, want the infra to be owned by a transparent community-aligned entity with diversified funding - like nix-community. If this were to be done in nix-community, we'd have to do some work upfront, like ensuring sufficiently smart scheduling to not jam other jobsets hosted by the organization. This would also probably increasing maintenance workload. This also raises questions about the scope of nix-community: how niche and how large of a project is acceptable? E.g. if nix-community does some GPU hardware stuff, why also not mobile, not IoT, not FPGA? Etc. If we decide that buying physical hardware is in-scope, we need to figure out how to manage the inventory and how to manage trust.

Despite all that, I do like the notion of doing this through nix-community, because it already up and running, it has a compatible structure, and it's already a recognized name.

In reply to @zowoq:matrix.org
I imagine that the amount and size of builds would make cachix or other cloud storage unfeasible. If it was only a dev cache could probably get away with just serving it off the CI master, if it was a proper public cache with non-trivial amount of users probably want a dedicated machine (or more than one if you want to keep the cache around for a while).

In reply to @ss:someonex.net
I think OpenCollective allows anonymous donations (in the sense of hiding the source from the public, but not from the project owners)

In reply to @ss:someonex.net
Regarding the ipfs message. That's not so much a substituter problem (an nginx instance with a local directory is more or less a substituter) as it is a build queue/hydra problem: afaiu we have no better solution for managing keys, than to copy outputs from all builders to a dedicated signer machine, nor do we have ready tools yet for doing this kind of cross-validation you describe. IPFS is more relevant for egress/distribution, which I'd say is a secondary concern right now. In either case, this is more of a research question, whereas with the CI we want to an immediate need with available tools