Nix Data Science

41 Members
18 Servers

Load older messages

3 Aug 2021
In reply to @tomberek:matrix.org
lunik1: that's a pattern I used Nix/Hydra for. Basically you have a set of "ingress"/"egress" derivations that may be impure (eg: fetch/store from S3) or pure. Then a chain of nix derivations that depend on each other. I defined a function to apply various transformations and map'd them to my list of ingress derivation. It was super nice for iteration, scaling up workers, cached results, experimenting with alternate pipelines. Way better and more productive than something like Airflow. I started to apply content-addressed derivations to them to do short-circuiting as well, it was still in progress for Hydra compatibility.
sounds somewhat similar to the way bionix models processing steps as nix functions, allowing you to easily map transformations over sets of inputs etc
@0x4a6f:matrix.org[0x4A6F]On that matter, has anyone got https://www.fluvio.io/ running?22:19:26
@vk3wtf:matrix.org@vk3wtf:matrix.orgof course i had cluster executing in mind as well since i had to make the computations work on slurm22:19:39
In reply to @vk3wtf:matrix.org
of course i had cluster executing in mind as well since i had to make the computations work on slurm
Do you have some configuration public for setting up slurm? I'm currently getting into HPC administration and I'm trying to get a slurm cluster up and running with nixops, so it'd be great to see what others use to set it up :)
@vk3wtf:matrix.org@vk3wtf:matrix.orgno i don't run the cluster with nix, i just submit jobs to it with nix22:34:25
@lunik1:lunik.onelunik1bionix looks nice but I gather is pretty tightly tied to bioinformatics?23:50:34
4 Aug 2021
@vk3wtf:matrix.org@vk3wtf:matrix.orgwell the library of tools is, but the general idea isn't00:48:11
@vk3wtf:matrix.org@vk3wtf:matrix.org at it's core it's just a collection of functions taking config -> inputs -> output (drvs), and building pipelines by composing them together 00:49:21
@vk3wtf:matrix.org@vk3wtf:matrix.orgthere's some small abstractions in bionix to allow for switching of the execution context, so that instead of the nix builder running the build it can be submitted to a cluster instead00:53:45
7 Aug 2021
@ss:someonex.netSomeone S

Ah, haven't visited this chat in a while...
I've just tried using nvidia containers with singularity and it won't see the GPUs - has anyone tried using singularity with gpus on NixOS?

nvidia-docker works, but singularity run --nv ~/Singularity/pytorch_21.07-py3.sif python -c "import torch; print(torch.cuda.is_available())" prints False and prior to that issues this warning:

WARNING: Unable to capture nv bind points: could not retrieve ld cache: could not execute ldconfig: fork/exec /sbin/ldconfig: no such file or directory
8 Aug 2021
@dearrude:nitro.chatEbrahim changed their display name from dearrude to Ebrahim.14:35:31
16 Aug 2021
@dearrude:nitro.chatEbrahim set a profile picture.14:41:12
4 Sep 2021
@phaebz:matrix.orgphaebz joined the room.19:26:30
8 Sep 2021
@ss:someonex.netSomeone S changed their profile picture.15:20:03
12 Sep 2021
@t_s_:matrix.orgTim Sears joined the room.22:24:10
13 Sep 2021
@vk3wtf:matrix.org@vk3wtf:matrix.org left the room.03:35:03
@jb:vk3.wtfvk3wtf joined the room.03:35:07
@jb:vk3.wtfvk3wtf changed their display name from vk3wtf to jbedo.03:35:23
@jb:vk3.wtfvk3wtf changed their display name from jbedo to vk3wtf.05:57:01
@mercierm:matrix.orgmercierm joined the room.11:42:08
17 Sep 2021
@tyberius_prime:matrix.orgTyberiusPrime joined the room.08:10:10
6 Oct 2021
@kamadorueda:matrix.orgkamadorueda joined the room.22:33:49
17 Oct 2021
@raitobezarius:matrix.orgraitobezarius joined the room.17:22:20
@raitobezarius:matrix.orgraitobezarius are there people who are able to install numpy using wheels with poetry2nix ? 17:22:43
@raitobezarius:matrix.orgraitobezarius * are there people who are able to install numpy using wheels with poetry2nix ? 17:22:45
@kamadorueda:matrix.orgkamadoruedaI don't use it, but I've seen issues on their bug tracker regarding that17:28:42
@raitobezarius:matrix.orgraitobezarius kamadorueda: do you remember a particular issue? I checked multiple times the issue tracker with no avail 17:29:09
@kamadorueda:matrix.orgkamadorueda (For anyone interested we continued the conversation at #python:nixos.org ) 17:46:45
@mars:jupiterbroadcasting.commars joined the room.19:02:55
20 Oct 2021
@emiller88:matrix.orgEdmundHello all, I was wondering if anyone had a project flake with micromamba or an example of mach.nix utilizing conda?16:23:38

There are no newer messages yet.

Back to Room List