While I don't have a huge amount of experience with this, one thing to keep in mind is that if you have huge datasets, they end up in the Nix store (unless you're doing something unusual) along with your output model. Your Nix store can really blow up in space. You may need to make sure you're garbage collecting frequently, but that may reduce the helpfulness of running training with Nix in the first place.

You'd also have to be careful with running builds on a machine that pushes results to a shared cache, since huge datasets and models could really clear out everything else in a cache.

You may also want to setup your Nix builders so they can see your GPUs. And fiddle with the max-jobs option so that two builds aren't trying to train in parallel.

In reply to @collinarnett:matrix.org
How do people deal with packages trying to download models at runtime to the nix store? This is more of a general question since I've noticed that a lot of data science libraries nowadays are just downloading dependencies at runtime.

In reply to @FRidh:matrix.org
I think all you can do is contact upstream to fix where they install to. If they need models, that could be fine, but then they should probably go to ~/.cache