!fXpAvneDgyJuYMZSwO:nixos.org

Nix Data Science

270 Members
57 Servers

You have reached the beginning of time (for this room).


SenderMessageTime
3 Sep 2023
@crtified:crtified.meCRTified
In reply to @jb:vk3.wtf
I use bionix pretty extensively, largest project currently involves processing ~90TiB of primary data
Does that dataset interact with the nix store in any way? Or is it kept separately? (Just asking, because I had some problems with putting just <30GB inside the store)
21:45:02
@crtified:crtified.meCRTifiedIt probably doesn't make sense to put it in the store, I guess 😄21:45:21
@jb:vk3.wtfjbedoIt does make sense and we do :)21:45:33
@crtified:crtified.meCRTifiedWow, that's surprising 😄21:46:15
@jb:vk3.wtfjbedoIt means we can cache some key intermediate products so we rarely need to revisit the primary inputs 21:46:26
@jb:vk3.wtfjbedoLarge store paths used to be an issue but mostly work pretty smoothly now as a lot of the memory bottlenecks have been removed21:49:18
@crtified:crtified.meCRTifiedThat makes sense. I really had the impression that large store entries still pose a lot of problems 🙂 Nice to know that this changed21:51:16
4 Sep 2023
@brodriguesco:matrix.orgBruno Rodriguesdoes that mean that you could retrieve intermediary outputs from the store to further Analyse in R?16:25:16
@brodriguesco:matrix.orgBruno Rodriguessay I create a ggplot does it get into the store, and could I look at it in R later?16:25:50
@crtified:crtified.meCRTifiedI mean, the store contains files after all. I don't see a reason why that shouldn't work16:26:07
@brodriguesco:matrix.orgBruno Rodriguesthat's essentially how targets works and I find it very useful 16:26:08
@brodriguesco:matrix.orgBruno Rodriguestrue, but I was wondering if bionix provides a mechanism for retrieving these files within an R session 16:26:49

Show newer messages


Back to Room ListRoom Version: 6