!fXpAvneDgyJuYMZSwO:nixos.org

Nix Data Science

288 Members
61 Servers

Load older messages


SenderMessageTime
23 Aug 2022
@carlthome:matrix.orgCarl Thomé
In reply to @jboy:utwente.io
These days I use a shell.nix with poetry and a venvHook, and then I just manage project dependencies (including jupyter) using poetry. It would be nice to do everything with nix packages, but inevitably I get to the point where I want to use something that isn't packaged.
Could you share an example of such a shell.nix? I'm interested in how to use a venvHook.
14:29:17
@carlthome:matrix.orgCarl Thomé
In reply to @carlthome:matrix.org
Could you share an example of such a shell.nix? I'm interested in how to use a venvHook.

I've been dabbling with jupyterWith but feeling that poetry2nix is easier when one just needs to get stuff done.

But I've only tried poetry2nix for making a library so far, not a notebook environment: https://github.com/carlthome/poetry2nix-test/blob/main/flake.nix

14:31:08
@jboy:utwente.ioJohn ✒️
14:31:57
@jboy:utwente.ioJohn ✒️ *
with import <nixpkgs> {};

mkShell {
  venvDir = "./VENV";
  buildInputs = [
    cairo
    graphviz
    zlib
    poetry
    python3.pkgs.venvShellHook
  ];
  postShellHook = ''
    export LD_LIBRARY_PATH=${stdenv.cc.cc.lib}/lib/:${cairo}/lib/:${zlib}/lib:$LD_LIBRARY_PATH
    poetry install
  '';
}
14:32:05
@jboy:utwente.ioJohn ✒️ *
with import <nixpkgs> {};

mkShell {
  venvDir = "./VENV";
  buildInputs = [
    cairo
    zlib
    poetry
    python3.pkgs.venvShellHook
  ];
  postShellHook = ''
    export LD_LIBRARY_PATH=${stdenv.cc.cc.lib}/lib/:${cairo}/lib/:${zlib}/lib:$LD_LIBRARY_PATH
    poetry install
  '';
}
14:32:26
@carlthome:matrix.orgCarl Thomé
In reply to @jboy:utwente.io
with import <nixpkgs> {};

mkShell {
  venvDir = "./VENV";
  buildInputs = [
    cairo
    zlib
    poetry
    python3.pkgs.venvShellHook
  ];
  postShellHook = ''
    export LD_LIBRARY_PATH=${stdenv.cc.cc.lib}/lib/:${cairo}/lib/:${zlib}/lib:$LD_LIBRARY_PATH
    poetry install
  '';
}
Thanks!
14:35:02
@flxai:matrix.orgFelixjboy: Sounds like a pragmatic solution if packaging is out of reach. Thanks for the snippet14:35:11
@flxai:matrix.orgFelixcarlthome: Have you also a nix file that uses jupyterWith? Where do you see friction with jupyterWith?14:36:08
@flxai:matrix.orgFelixAnd thanks for the poetry2nix file14:36:19
@carlthome:matrix.orgCarl Thomé
In reply to @flxai:matrix.org
carlthome: Have you also a nix file that uses jupyterWith? Where do you see friction with jupyterWith?

TBH not sure what I'm doing really. 🫡

Figured it could be convenient to have a set of regular ipykernels (one for PyTorch, one for scikit-learn, etc.), ready for playing around with ML modelling ideas.

Thus I started just throwing in them as devShells in my ongoing home flake to learn more about jupyterWith.

Here's a wip commit:

https://github.com/carlthome/dotfiles/pull/3/files#diff-206b9ce276ab5971a2489d75eb1b12999d4bf3843b7988cbe8d687cfde61dea0R57

14:43:54
@flxai:matrix.orgFelixYes I feel you. There are many solutions to this problem. Will try those posted. Thanks again14:54:50
24 Aug 2022
@cafkafk:matrix.cafkafk.comcafkafk joined the room.19:15:39
@tinybronca:sibnsk.netunderpantsgnome changed their display name from tinybronca to underpantsgnome.23:00:44
25 Aug 2022
@entheogenesis:matrix.orgentheogenesis joined the room.17:46:09
30 Aug 2022
@sents:matrix.orgsents joined the room.13:42:27
@aruzeta:matrix.orgaru joined the room.14:42:54
@aruzeta:matrix.orgaru left the room.14:56:16
@hinklejd:matrix.orghacobjinkle joined the room.16:51:54
5 Sep 2022
@kotatsuyaki:matrix.orgkotatsuyaki joined the room.05:39:36
@michelvasconcelos:matrix.orgmichelvasconcelos joined the room.18:21:51
6 Sep 2022
@hinklejd:matrix.orghacobjinkleNix is attractive for reproducibility and provenance tracking. However, for ML workflows one major limitation is that all data produced as part of a derivation resides in /nix/store. It would be great to be able to define flakes for large datasets and use them as inputs to other programs/flakes. For example define a flake for ImageNet and then get train and test splits as outputs of those downstream flakes. Currently such a train/test split would require duplication of the original dataset twice into /nix/store if I'm not mistaken.15:34:39
@hinklejd:matrix.orghacobjinkleapologies if there is an existing solution to this kind of data handling with nix that I'm unaware of15:35:12
@hinklejd:matrix.orghacobjinklei guess the splitting could of course be done with symlinks. main question is then is there any hope of handling large datasets at rest on some storage outside /nix15:39:06
@FRidh:matrix.orgFRidhNix master supports impure derivations. Impure derivations have network access, so you could write your result somewhere else. I have not tried this yet, but I imagine this going to be very useful also in CI/CD.17:04:49
@FRidh:matrix.orgFRidhhttps://github.com/NixOS/nix/pull/622717:05:19
@hinklejd:matrix.orghacobjinklethanks. I currently use --impure on non-NixOS with nixGL to get cuda working anyway, so that may be the ticket17:07:24
@FRidh:matrix.orgFRidhthat's not the same though, that's impure evaluation17:08:40
@hinklejd:matrix.orghacobjinklei see that now. should've read before replying..17:09:23
7 Sep 2022
@ss:someonex.netSomeone (UTC+3)

Has anyone got a working example of micromamba environment?
I've tried using it in a bulidFHSUserEnv(Bubblewrap) with this in /etc/profile:

    eval "$(micromamba shell hook --shell=bash)"
    micromamba activate

...but it keeps segfaulting later when I prompt micromamba activate $envName

What helps is running the eval "$(...)" in the shell manually

12:43:40
15 Sep 2022
@m_algery:leohoo.xyzm_algery joined the room.12:10:46

Show newer messages


Back to Room ListRoom Version: 6