| 8 Jun 2021 |
|  Jez (he/him) 🌦️ changed their profile picture. | 09:41:33 |
| 10 Jun 2021 |
|  Church joined the room. | 04:29:57 |
| 11 Jun 2021 |
|  Manuel Bärenz joined the room. | 10:11:13 |
| 22 Jun 2021 |
|  Regnat joined the room. | 13:11:42 |
| 26 Jun 2021 |
|  @grahamc:nixos.orgchanged room power levels. | 01:18:35 |
| @grahamc:nixos.org invited  NixOS Moderation Bot. | 01:23:59 |
| NixOS Moderation Bot joined the room. | 01:24:00 |
| 27 Jun 2021 |
|  tomberek joined the room. | 04:25:51 |
| 30 Jun 2021 |
|  kraftnix joined the room. | 17:35:17 |
| 1 Jul 2021 |
|  avx3 joined the room. | 15:58:42 |
| 5 Jul 2021 |
|  spacesbot - keeps a log of public NixOS channels joined the room. | 19:21:44 |
| spacesbot - keeps a log of public NixOS channels | 19:49:33 |
| 6 Jul 2021 |
| spacesbot - keeps a log of public NixOS channels changed their display name from spacesbot to spacesbot - keeps a log of public NixOS channels. | 22:11:46 |
| 9 Jul 2021 |
|  junji hashimoto joined the room. | 00:19:12 |
| 14 Jul 2021 |
|  Stephan Sahm joined the room. | 09:13:14 |
| 21 Jul 2021 |
|  CRTified joined the room. | 10:20:18 |
| 23 Jul 2021 |
| Room Avatar Renderer. | 23:22:41 |
| 29 Jul 2021 |
|  Ebrahim joined the room. | 19:22:32 |
| 3 Aug 2021 |
|  lunik1 left the room. | 11:23:58 |
| lunik1 joined the room. | 11:24:06 |
| lunik1 | Any recommendations for data pipeline/toolkits that integrate well with nix? | 11:31:51 |
 jbedo | to do what? i use nix with a thin layer on top to manage bioinformatics pipelines | 12:03:59 |
| tomberek | jbedo: do you use bionix? or is it another thing? | 13:12:47 |
| lunik1 | Currently I just have a bunch of python scripts I execute in a given order, but I'm looking for something that would help me formalise that order, easily extend/swap out parts of those pipelines, and help with deployment | 15:01:59 |
| tomberek | lunik1: that's a pattern I used Nix/Hydra for. Basically you have a set of "ingress"/"egress" derivations that may be impure (eg: fetch/store from S3) or pure. Then a chain of nix derivations that depend on each other. I defined a function to apply various transformations and map'd them to my list of ingress derivation. It was super nice for iteration, scaling up workers, cached results, experimenting with alternate pipelines. Way better and more productive than something like Airflow. I started to apply content-addressed derivations to them to do short-circuiting as well, it was still in progress for Hydra compatibility. | 19:36:25 |
| lunik1 | Damn that sounds awesome, any of this open source? | 19:37:47 |
| tomberek | No. My plan is to capture the idea, organize it a bit better, and have that be open source. I've heard of a few people re-inventing this a few times, so I want extract out the common portions and perhaps provide a "flow-library" or something to make it easier to put together. | 19:39:30 |
| tomberek | I'd be happy to collaborate on it. | 19:39:45 |
| lunik1 | Was that all batch processing or could you handle streaming data too? | 19:39:48 |
| tomberek | It was not streaming (in the sense that you just shoved in requests on one and and results popped out the other), but it ended up having lower latency than our streaming solution.... so....... But that was because of Hydra, you can use the same "flow-library" and use a different evaluator/build system to get something more streaming | 19:41:46 |
