| 1 Sep 2023 |
 Bruno Rodrigues | try to post a minimal reproducible example of what you're looking for | 10:33:44 |
 maupind | Bruno Rodrigues: awesome, thank you! This is definitely a good start and I'll explore this and before trying Posit | 10:37:57 |
| 3 Sep 2023 |
| Bruno Rodrigues | friends, what do you use to build pipelines that integrate will with Nix? something that would let you build an environment and run a pipeline as simple as possible? me, coming from R I'm adding the command to run the pipeline in the shellHook of a shell (typically targets::tar_make) | 14:04:29 |
| Bruno Rodrigues | sou dropping in the shell starts the pipeline and then let's me explore the results | 14:05:00 |
| Bruno Rodrigues | I would imagine que there's likely a way to use nix itself to do it | 14:09:20 |
 CRTified | Maybe check out https://github.com/PapenfussLab/bionix - it appears to at least partially match your description (although I don't have any experience with it) | 14:10:17 |
| Bruno Rodrigues | yep I've taken a look at it already 😁 | 14:11:22 |
| CRTified | Can't really help further than that, I'm still pondering whether I should migrate our hpc cluster to NixOS, but my contract ends soon and I likely won't have enough time | 14:13:14 |
| CRTified | * Can't really help further than that, I'm still only pondering whether I should migrate our hpc cluster to NixOS, but my contract ends soon and I likely won't have enough time | 14:13:27 |
 jbedo | I use bionix pretty extensively, largest project currently involves processing ~90TiB of primary data | 21:43:06 |
| CRTified | In reply to @jb:vk3.wtf
I use bionix pretty extensively, largest project currently involves processing ~90TiB of primary data Does that dataset interact with the nix store in any way? Or is it kept separately? (Just asking, because I had some problems with putting just <30GB inside the store) | 21:45:02 |
| CRTified | It probably doesn't make sense to put it in the store, I guess 😄 | 21:45:21 |
| jbedo | It does make sense and we do :) | 21:45:33 |
| CRTified | Wow, that's surprising 😄 | 21:46:15 |
| jbedo | It means we can cache some key intermediate products so we rarely need to revisit the primary inputs | 21:46:26 |
| jbedo | Large store paths used to be an issue but mostly work pretty smoothly now as a lot of the memory bottlenecks have been removed | 21:49:18 |
| CRTified | That makes sense. I really had the impression that large store entries still pose a lot of problems 🙂 Nice to know that this changed | 21:51:16 |
| 4 Sep 2023 |
| Bruno Rodrigues | does that mean that you could retrieve intermediary outputs from the store to further Analyse in R? | 16:25:16 |
| Bruno Rodrigues | say I create a ggplot does it get into the store, and could I look at it in R later? | 16:25:50 |
| CRTified | I mean, the store contains files after all. I don't see a reason why that shouldn't work | 16:26:07 |
| Bruno Rodrigues | that's essentially how targets works and I find it very useful | 16:26:08 |
| Bruno Rodrigues | true, but I was wondering if bionix provides a mechanism for retrieving these files within an R session | 16:26:49 |
| Bruno Rodrigues | and also suppose I fit a model, and would like to save this model | 16:31:35 |
| Bruno Rodrigues | with targets this model gets serialized and saved for later retrieval in an r session | 16:32:06 |
| Bruno Rodrigues | does nix/bionix serialize intermediary outputs ? | 16:32:46 |
| CRTified | I think jbedo could help a bit more for the bionix question, but for additional outputs, I'd actually check the outputs argument for mkDerivation | 16:33:08 |
| CRTified | The typical usecase are distinct outputs for libs, bin and so on, but I think it would fit that application nicely as well | 16:33:37 |
| jbedo | In reply to @brodriguesco:matrix.org
with targets this model gets serialized and saved for later retrieval in an r session there's no knowledge of bionix at the application level, for R i use the builtin serialisation like save() and writeRDS(), and as CRTified said multiple outputs are handy for dealilng with several outputs produced simultaneously | 22:27:20 |
| 5 Sep 2023 |
| Bruno Rodrigues | very nice I'll have to play around with it | 07:18:04 |
|  ri-char left the room. | 15:55:18 |
