In reply to @tomberek:matrix.org
lunik1: that's a pattern I used Nix/Hydra for. Basically you have a set of "ingress"/"egress" derivations that may be impure (eg: fetch/store from S3) or pure. Then a chain of nix derivations that depend on each other. I defined a function to apply various transformations and map'd them to my list of ingress derivation. It was super nice for iteration, scaling up workers, cached results, experimenting with alternate pipelines. Way better and more productive than something like Airflow. I started to apply content-addressed derivations to them to do short-circuiting as well, it was still in progress for Hydra compatibility.

In reply to @vk3wtf:matrix.org
of course i had cluster executing in mind as well since i had to make the computations work on slurm

Ah, haven't visited this chat in a while...
I've just tried using nvidia containers with singularity and it won't see the GPUs - has anyone tried using singularity with gpus on NixOS?

nvidia-docker works, but singularity run --nv ~/Singularity/pytorch_21.07-py3.sif python -c "import torch; print(torch.cuda.is_available())" prints False and prior to that issues this warning:

WARNING: Unable to capture nv bind points: could not retrieve ld cache: could not execute ldconfig: fork/exec /sbin/ldconfig: no such file or directory