| 16 Jan 2024 |
 benoitdr | I'm not sure how you are working. Personally, I use nix-shell to package all the development environment (including python modules and non-python packages). Here is an example for jupyter lab :
let
nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/04220ed6763637e5899980f98d5c8424b1079353.tar.gz";
pkgs = import nixpkgs { config = {}; overlays = []; };
in
pkgs.mkShell {
packages = with pkgs; [
(python310.withPackages (ps: with ps; [
numpy
pandas
matplotlib
scipy
jupyterlab
ipympl
jupyterlab-git
]))
git
];
shellHook = ''
jupyter lab
'';
}
Note that unless you use nix-shell --pure, you don't need to add git if it is alrready present at OS level, although it's probability better to add it anyway for portability
| 12:34:26 |
| 17 Jan 2024 |
|  andredornas joined the room. | 13:51:45 |
| 18 Jan 2024 |
 fizihcyst | Hi, is anyone here using julia? I see that recently a pr was merged to nixpkgs to build julia.withPackages similar to python. This is fine for simple scripts, but does anyone have recommendations for using nix with a julia Project.toml/Manifest.toml? Maybe something similar to poetry2nix? I see two julia2nix repos, but had trouble getting them to work. A working example julia project+flake would be helpful. | 17:08:43 |
| 22 Jan 2024 |
|  Gregory M. Kapfhammer joined the room. | 16:59:42 |
| Gregory M. Kapfhammer | Hello, does anyone have a good example of how to get Quarto to work on NixOS?
I created this site using Quarto on Arch Linux: https://github.com/gkapfham/www.gregorykapfhammer.com
This configuration allows me to use Poetry to manage the project's dependencies.
When I am in the poetry shell I can use quarto and it finds all of the project's dependencies in the virtualenv when I run it on Arch Linux.
However, when I use NixOS the quarto program installed through nix packages does not seem to pass along the dependencies in the virtual environment.
I am glad to share more details. With that said, does anyone have a quick idea as to what I should try next? Thanks! | 18:27:19 |
|  phiadaarr joined the room. | 21:19:10 |
| 23 Jan 2024 |
|  bcdarwin joined the room. | 22:54:56 |
| 25 Jan 2024 |
 trexd | What are people's thoughts around putting data in the nix store? Only small stuff? Large datasets too? Only in specific situations? | 16:25:09 |
 CRTified | Generally a nice thing, although painful for larger files. Not directly related to data science, but it was a pain to get vivado (fpga IDE/tool chain) into the store, and that's only like 25GiB worth of data | 16:34:48 |
| trexd | Ok thats around the dataset size that I'm dealing with but I can decrompress it before training so fitting it in the store compressed should make things easier. | 16:44:30 |
| 26 Jan 2024 |
| benoitdr | Is everyone OK with that ? I guess uploading large datasets to the nix store will increase the nix store hosting costs (on S3 ?). Another option could be to store the datasets in dedicated platforms (HugggingFace, Kaggle, ....) and store pointers in the nix store. | 09:03:33 |
| CRTified | In reply to @benoitdr:matrix.org
Is everyone OK with that ? I guess uploading large datasets to the nix store will increase the nix store hosting costs (on S3 ?). Another option could be to store the datasets in dedicated platforms (HugggingFace, Kaggle, ....) and store pointers in the nix store. Depending on how you point to the dataset, it will end up in the nix store after pulling | 11:15:51 |
| CRTified | Might require to break the sandboxing to allow network access | 11:16:10 |
 maupind | Does anyone have any experience overwriting python dependencies? I'm trying to get a project with botorch/ax working, but they have the package linear operator as a dependency which is failing due to its typeguard dependency being far behind. I feel like I am close with a post patch override but something is missing | 15:27:38 |
 raitobezarius | In reply to @benoitdr:matrix.org
Is everyone OK with that ? I guess uploading large datasets to the nix store will increase the nix store hosting costs (on S3 ?). Another option could be to store the datasets in dedicated platforms (HugggingFace, Kaggle, ....) and store pointers in the nix store. OK with what? | 15:32:39 |
| raitobezarius | We were not talking about cache.nixos.org, were we? | 15:32:58 |
| benoitdr | In reply to @raitobezarius:matrix.org
OK with what? OK with storing datasets in the nix store | 15:33:30 |
| raitobezarius | Well whether you upload it to the Nix store or to some random platform, what does that change? | 15:34:11 |
| raitobezarius | It needs to be stored somewhere? | 15:34:16 |
| benoitdr | In reply to @raitobezarius:matrix.org
Well whether you upload it to the Nix store or to some random platform, what does that change? the one who is paying for the storage ;-) There is no free lunch ... | 15:34:54 |
| raitobezarius | Again, we are not talking about cache.nixos.org, are we? | 15:35:28 |
| raitobezarius | Like even if you have pointers inside your remote storage, that does not change a lot that you are going to store a large buffer at some point somewhere, either in memory or on disk | 15:35:53 |
| benoitdr | In reply to @raitobezarius:matrix.org
Again, we are not talking about cache.nixos.org, are we? that's what I understood what maybe I got it wrong | 15:36:08 |
| raitobezarius | "The Nix store" is not cache.nixos.org, right? | 15:36:32 |
| benoitdr | In reply to @raitobezarius:matrix.org
Again, we are not talking about cache.nixos.org, are we? * that's what I understood but maybe I got it wrong | 15:36:44 |
| raitobezarius | Everyone who is using Nix has "the Nix store", in their local filesystem, it is in /nix/store | 15:36:46 |
| raitobezarius | But yes, storing datasets in cache.nixos.org is mostly out of the question, at the moment, for the cost reasons you mentioned | 15:37:12 |
| raitobezarius | Storing datasets in your own Nix store seems an interesting question to me though | 15:37:20 |
| raitobezarius | Compared to… store it locally on-disk, mount it from a remote location and buffer it, etc. | 15:37:44 |
| benoitdr | fully agree on that ... trexd , Can you clarify your request ? | 15:38:27 |
