Nix Data Science | 296 Members | |
| 62 Servers |
| Sender | Message | Time |
|---|---|---|
| 15 Aug 2022 | ||
| 23:19:31 | ||
| 16 Aug 2022 | ||
| 11:48:20 | ||
| Hello :) | 11:51:31 | |
| I am new to NixOS and wondering what's a good way to set up Jupyter. Are there opinions on JupyterWith or maybe some general recommendations? | 11:52:27 | |
| 17 Aug 2022 | ||
| 20:24:12 | ||
| 21 Aug 2022 | ||
| 09:25:58 | ||
| 23 Aug 2022 | ||
| 00:50:04 | ||
In reply to @flxai:matrix.orgThese days I use a shell.nix with poetry and a venvHook, and then I just manage project dependencies (including jupyter) using poetry. It would be nice to do everything with nix packages, but inevitably I get to the point where I want to use something that isn't packaged. | 13:36:28 | |
In reply to @jboy:utwente.ioCould you share an example of such a shell.nix? I'm interested in how to use a venvHook. | 14:29:17 | |
In reply to @carlthome:matrix.org I've been dabbling with jupyterWith but feeling that poetry2nix is easier when one just needs to get stuff done. But I've only tried poetry2nix for making a library so far, not a notebook environment: https://github.com/carlthome/poetry2nix-test/blob/main/flake.nix | 14:31:08 | |
| 14:31:57 | |
*
| 14:32:05 | |
*
| 14:32:26 | |
In reply to @jboy:utwente.ioThanks! | 14:35:02 | |
| jboy: Sounds like a pragmatic solution if packaging is out of reach. Thanks for the snippet | 14:35:11 | |
| carlthome: Have you also a nix file that uses jupyterWith? Where do you see friction with jupyterWith? | 14:36:08 | |
| And thanks for the poetry2nix file | 14:36:19 | |
In reply to @flxai:matrix.org TBH not sure what I'm doing really. 🫡 Figured it could be convenient to have a set of regular ipykernels (one for PyTorch, one for scikit-learn, etc.), ready for playing around with ML modelling ideas. Thus I started just throwing in them as devShells in my ongoing home flake to learn more about jupyterWith. Here's a wip commit: https://github.com/carlthome/dotfiles/pull/3/files#diff-206b9ce276ab5971a2489d75eb1b12999d4bf3843b7988cbe8d687cfde61dea0R57 | 14:43:54 | |
| Yes I feel you. There are many solutions to this problem. Will try those posted. Thanks again | 14:54:50 | |
| 24 Aug 2022 | ||
| 19:15:39 | ||
| 23:00:44 | ||
| 25 Aug 2022 | ||
| 17:46:09 | ||
| 30 Aug 2022 | ||
| 13:42:27 | ||
| 14:42:54 | ||
| 14:56:16 | ||
| 16:51:54 | ||
| 5 Sep 2022 | ||
| 05:39:36 | ||
| 18:21:51 | ||
| 6 Sep 2022 | ||
| Nix is attractive for reproducibility and provenance tracking. However, for ML workflows one major limitation is that all data produced as part of a derivation resides in /nix/store. It would be great to be able to define flakes for large datasets and use them as inputs to other programs/flakes. For example define a flake for ImageNet and then get train and test splits as outputs of those downstream flakes. Currently such a train/test split would require duplication of the original dataset twice into /nix/store if I'm not mistaken. | 15:34:39 | |
| apologies if there is an existing solution to this kind of data handling with nix that I'm unaware of | 15:35:12 | |