| 6 Jul 2022 |
 jbedo | https://github.com/NixOS/nix/pull/6227 | 06:55:00 |
| jbedo | i haven't really thought about this in the context of data pipelines, just remember seeing it pass my notifications | 06:55:36 |
 cdepillabout | While I don't have a huge amount of experience with this, one thing to keep in mind is that if you have huge datasets, they end up in the Nix store (unless you're doing something unusual) along with your output model. Your Nix store can really blow up in space. You may need to make sure you're garbage collecting frequently, but that may reduce the helpfulness of running training with Nix in the first place.
You'd also have to be careful with running builds on a machine that pushes results to a shared cache, since huge datasets and models could really clear out everything else in a cache.
You may also want to setup your Nix builders so they can see your GPUs. And fiddle with the max-jobs option so that two builds aren't trying to train in parallel.
| 07:45:28 |
| cdepillabout | If you're trying to productionize something, it may be easier to start with a system that already has answers for all this type of stuff (some MLOps service or something?), rather than rolling your own with Nix. | 07:55:13 |
| 9 Jul 2022 |
|  betaboon changed their profile picture. | 11:33:12 |
| 12 Jul 2022 |
|  pbsds joined the room. | 22:55:46 |
| 26 Jul 2022 |
|  underpantsgnome changed their display name from tinybronca to tailrec. | 14:39:42 |
| underpantsgnome changed their display name from tailrec to tinybronca. | 15:40:54 |
|  IPK joined the room. | 18:32:31 |
| 27 Jul 2022 |
|  Collin Arnett joined the room. | 20:39:42 |
| 1 Aug 2022 |
|  better_sleeping joined the room. | 09:06:29 |
| better_sleeping left the room. | 09:06:44 |
| Collin Arnett | How do people deal with packages trying to download models at runtime to the nix store? This is more of a general question since I've noticed that a lot of data science libraries nowadays are just downloading dependencies at runtime. | 18:18:39 |
| 2 Aug 2022 |
 FRidh | In reply to @collinarnett:matrix.org
How do people deal with packages trying to download models at runtime to the nix store? This is more of a general question since I've noticed that a lot of data science libraries nowadays are just downloading dependencies at runtime. I think all you can do is contact upstream to fix where they install to. If they need models, that could be fine, but then they should probably go to ~/.cache | 09:26:59 |
| Collin Arnett | In reply to @FRidh:matrix.org
I think all you can do is contact upstream to fix where they install to. If they need models, that could be fine, but then they should probably go to ~/.cache Yeah I think this is probably the best solution. Thank you :) | 17:16:04 |
| 6 Aug 2022 |
|  Yuu Yin changed their display name from yuu to yuu[m]. | 17:42:49 |
| Yuu Yin changed their display name from yuu[m] to yuu. | 20:35:43 |
| 13 Aug 2022 |
| Yuu Yin changed their display name from yuu to yuu[m]. | 03:15:45 |
| Yuu Yin changed their display name from yuu[m] to yuu. | 05:06:14 |
| 15 Aug 2022 |
| pbsds changed their display name from pederbs to pbsds. | 23:19:31 |
| 16 Aug 2022 |
|  Felix joined the room. | 11:48:20 |
| Felix | Hello :) | 11:51:31 |
| Felix | I am new to NixOS and wondering what's a good way to set up Jupyter. Are there opinions on JupyterWith or maybe some general recommendations? | 11:52:27 |
| 17 Aug 2022 |
|  greaka joined the room. | 20:24:12 |
| 21 Aug 2022 |
| greaka changed their display name from greaka ⚡️ to greaka . | 09:25:58 |
| 23 Aug 2022 |
|  Echo joined the room. | 00:50:04 |
 John ✒️ | In reply to @flxai:matrix.org
I am new to NixOS and wondering what's a good way to set up Jupyter. Are there opinions on JupyterWith or maybe some general recommendations? These days I use a shell.nix with poetry and a venvHook, and then I just manage project dependencies (including jupyter) using poetry. It would be nice to do everything with nix packages, but inevitably I get to the point where I want to use something that isn't packaged. | 13:36:28 |
 Carl Thomé | In reply to @jboy:utwente.io
These days I use a shell.nix with poetry and a venvHook, and then I just manage project dependencies (including jupyter) using poetry. It would be nice to do everything with nix packages, but inevitably I get to the point where I want to use something that isn't packaged. Could you share an example of such a shell.nix? I'm interested in how to use a venvHook. | 14:29:17 |
| Carl Thomé | In reply to @carlthome:matrix.org
Could you share an example of such a shell.nix? I'm interested in how to use a venvHook. I've been dabbling with jupyterWith but feeling that poetry2nix is easier when one just needs to get stuff done.
But I've only tried poetry2nix for making a library so far, not a notebook environment: https://github.com/carlthome/poetry2nix-test/blob/main/flake.nix
| 14:31:08 |
| John ✒️ |
| 14:31:57 |
