| 29 Jul 2021 |
|  Ebrahim joined the room. | 19:22:32 |
| 3 Aug 2021 |
|  lunik1 left the room. | 11:23:58 |
| lunik1 joined the room. | 11:24:06 |
| lunik1 | Any recommendations for data pipeline/toolkits that integrate well with nix? | 11:31:51 |
 jbedo | to do what? i use nix with a thin layer on top to manage bioinformatics pipelines | 12:03:59 |
 tomberek | jbedo: do you use bionix? or is it another thing? | 13:12:47 |
| lunik1 | Currently I just have a bunch of python scripts I execute in a given order, but I'm looking for something that would help me formalise that order, easily extend/swap out parts of those pipelines, and help with deployment | 15:01:59 |
| tomberek | lunik1: that's a pattern I used Nix/Hydra for. Basically you have a set of "ingress"/"egress" derivations that may be impure (eg: fetch/store from S3) or pure. Then a chain of nix derivations that depend on each other. I defined a function to apply various transformations and map'd them to my list of ingress derivation. It was super nice for iteration, scaling up workers, cached results, experimenting with alternate pipelines. Way better and more productive than something like Airflow. I started to apply content-addressed derivations to them to do short-circuiting as well, it was still in progress for Hydra compatibility. | 19:36:25 |
| lunik1 | Damn that sounds awesome, any of this open source? | 19:37:47 |
| tomberek | No. My plan is to capture the idea, organize it a bit better, and have that be open source. I've heard of a few people re-inventing this a few times, so I want extract out the common portions and perhaps provide a "flow-library" or something to make it easier to put together. | 19:39:30 |
| tomberek | I'd be happy to collaborate on it. | 19:39:45 |
| lunik1 | Was that all batch processing or could you handle streaming data too? | 19:39:48 |
| tomberek | It was not streaming (in the sense that you just shoved in requests on one and and results popped out the other), but it ended up having lower latency than our streaming solution.... so....... But that was because of Hydra, you can use the same "flow-library" and use a different evaluator/build system to get something more streaming | 19:41:46 |
| lunik1 | So you could effectively run in "real time"? With some sort of API? | 20:09:37 |
| tomberek | potentially, but that comes with more complexity and requirements that i'd like to avoid (or at least be agnostic about) for now | 20:16:20 |
| lunik1 | For my application I would need some functionality like that, and nix doesn't seem obviously suited to it | 20:20:35 |
| jbedo | In reply to @tomberek:matrix.org
jbedo: do you use bionix? or is it another thing? yeah bionix | 22:17:54 |
| jbedo | In reply to @tomberek:matrix.org
lunik1: that's a pattern I used Nix/Hydra for. Basically you have a set of "ingress"/"egress" derivations that may be impure (eg: fetch/store from S3) or pure. Then a chain of nix derivations that depend on each other. I defined a function to apply various transformations and map'd them to my list of ingress derivation. It was super nice for iteration, scaling up workers, cached results, experimenting with alternate pipelines. Way better and more productive than something like Airflow. I started to apply content-addressed derivations to them to do short-circuiting as well, it was still in progress for Hydra compatibility. sounds somewhat similar to the way bionix models processing steps as nix functions, allowing you to easily map transformations over sets of inputs etc | 22:19:03 |
 [0x4A6F] | On that matter, has anyone got https://www.fluvio.io/ running? | 22:19:26 |
| jbedo | of course i had cluster executing in mind as well since i had to make the computations work on slurm | 22:19:39 |
 CRTified (old handle) | In reply to @vk3wtf:matrix.org
of course i had cluster executing in mind as well since i had to make the computations work on slurm Do you have some configuration public for setting up slurm? I'm currently getting into HPC administration and I'm trying to get a slurm cluster up and running with nixops, so it'd be great to see what others use to set it up :) | 22:25:17 |
| jbedo | no i don't run the cluster with nix, i just submit jobs to it with nix | 22:34:25 |
| lunik1 | bionix looks nice but I gather is pretty tightly tied to bioinformatics? | 23:50:34 |
| 4 Aug 2021 |
| jbedo | well the library of tools is, but the general idea isn't | 00:48:11 |
| jbedo | at it's core it's just a collection of functions taking config -> inputs -> output (drvs), and building pipelines by composing them together | 00:49:21 |
| jbedo | there's some small abstractions in bionix to allow for switching of the execution context, so that instead of the nix builder running the build it can be submitted to a cluster instead | 00:53:45 |
| 7 Aug 2021 |
 SomeoneSerge (Way down Hadestown) | Ah, haven't visited this chat in a while...
I've just tried using nvidia containers with singularity and it won't see the GPUs - has anyone tried using singularity with gpus on NixOS?
nvidia-docker works, but singularity run --nv ~/Singularity/pytorch_21.07-py3.sif python -c "import torch; print(torch.cuda.is_available())" prints False and prior to that issues this warning:
WARNING: Unable to capture nv bind points: could not retrieve ld cache: could not execute ldconfig: fork/exec /sbin/ldconfig: no such file or directory
| 21:10:05 |
| 8 Aug 2021 |
| Ebrahim changed their display name from dearrude to Ebrahim. | 14:35:31 |
| 16 Aug 2021 |
| Ebrahim set a profile picture. | 14:41:12 |
| 4 Sep 2021 |
|  phaebz joined the room. | 19:26:30 |
