In reply to @benoitdr:matrix.org

Indeed a bit of documentation would help ;-)
Looking at cudatoolkit sources, I can see that it's importing many many things I don't need, so I would be happy to get rid of it.
Unfortunately, if I replace it by cuda_nvcc, cuda_cudart and libcublas, ctransformers doesn't build anymore.

-- Using CUDA architectures: 52;61;70
-- Unable to find cuda_runtime.h in "/nix/store/p8058x6fpdlw7hy72qsqn41qhllqncgm-cuda_nvcc-11.8.89/include" for CUDAToolkit_INCLUDE_DIR.
-- Unable to find cublas_v2.h in either "" or "/nix/math_libs/include"
-- Could NOT find CUDAToolkit (missing: CUDAToolkit_INCLUDE_DIR) (found version "11.8.89")
CMake Warning at CMakeLists.txt:163 (message):
  cuBLAS not found

Looking at CMakeLists.txt :

if (CT_CUBLAS)
    find_package(CUDAToolkit)

So it seems there is a problem with the CUDAToolkit_INCLUDE_DIR env variable.
Not sure it's related, but looking at https://cmake.org/cmake/help/latest/module/FindCUDAToolkit.html, that function is setting a CUDAToolkit_INCLUDE_DIRS (with extra S)

In reply to @benoitdr:matrix.org

Indeed a bit of documentation would help ;-)
Looking at cudatoolkit sources, I can see that it's importing many many things I don't need, so I would be happy to get rid of it.
Unfortunately, if I replace it by cuda_nvcc, cuda_cudart and libcublas, ctransformers doesn't build anymore.

-- Using CUDA architectures: 52;61;70
-- Unable to find cuda_runtime.h in "/nix/store/p8058x6fpdlw7hy72qsqn41qhllqncgm-cuda_nvcc-11.8.89/include" for CUDAToolkit_INCLUDE_DIR.
-- Unable to find cublas_v2.h in either "" or "/nix/math_libs/include"
-- Could NOT find CUDAToolkit (missing: CUDAToolkit_INCLUDE_DIR) (found version "11.8.89")
CMake Warning at CMakeLists.txt:163 (message):
  cuBLAS not found

Looking at CMakeLists.txt :

if (CT_CUBLAS)
    find_package(CUDAToolkit)

So it seems there is a problem with the CUDAToolkit_INCLUDE_DIR env variable.
Not sure it's related, but looking at https://cmake.org/cmake/help/latest/module/FindCUDAToolkit.html, that function is setting a CUDAToolkit_INCLUDE_DIRS (with extra S)

{ config, lib, pkgs, ... }:

{

services.jupyterhub = {

enable = true;
jupyterhubEnv = pkgs.python311.withPackages (p: with p; [
    jupyterhub
    jupyterhub-systemdspawner
    pip
    ipython
    jupyter
    notebook
    ipykernel
  ]);
jupyterlabEnv = pkgs.python311.withPackages (p: with p; [
    jupyterhub
    jupyterlab
    pip
    ipython
    jupyter
    notebook
    ipykernel
  ]);
kernels = {
  python311 = let
    env = (pkgs.python311.withPackages (python311Packages: with python311Packages; [
      ipykernel
      pandas
      scikit-learn
      numpy
      seaborn
      matplotlib
      scipy
    ]));
  in {
    displayName = "Python 3 for ML";
    argv = [
      "${env.interpreter}"
      "-m"
      "ipykernel_launcher"
      "-f"
      "{connection_file}"
    ];
    language = "python";
    logo32 = "${env.sitePackages}/ipykernel/resources/logo-32x32.png";
    logo64 = "${env.sitePackages}/ipykernel/resources/logo-64x64.png";
  };
  R = let
    env = (pkgs.rWrapper.override {
      packages = with pkgs.rPackages; [
        IRkernel
        tidyverse
        caret
        randomForest
        tidymodels
        purrr
        shiny
        data_table
        jsonlite
      ];
    });
  in {
    displayName = "R for ML";
    argv = [
      "/nix/store/pqynpqccbr41zz3pg844qiq2h87mxhwx-R-4.3.2-wrapper/bin/R"
      "--slave"
      "-e"
      "IRkernel::main()"
      "--args"
      "{connection_file}"
    ];
    language = "R";
    logo32 = "${env.sitePackages}/IRKernel/resources/logo-32x32.png";
    logo64 = "${env.sitePackages}/IRkernel/resources/logo-64x64.png";
  };
};

};

}

In reply to @gambrose:matrix.org
Hi all. New here and am looking for some help. I am trying to get a persistent jupyterhub/lab up and running on nixos. I have made progress, but stuck on two items. First, is it possible in the approach I am using to add jupyterlab extensions declaratively? I saw that its possible if I build a shell to use, but I would rather not take that approach. Second, while I got a python env up and running with some basic data science packages to try out, but I can't get an R to work. Console won't connect to the kernel, even after multiple attempts, including finding and using the full path to the R bin. Pasting my code below, if anyone has pointers who may have tried this. Thanks!

In reply to @benoitdr:matrix.org
For the extensions, can't you just add them like other packages ? See jupyterlab-widgets, jupyterlab-git, jupyterlab-lsp, ...

In reply to @benoitdr:matrix.org
For the extensions, can't you just add them like other packages ? See jupyterlab-widgets, jupyterlab-git, jupyterlab-lsp, ...

Yes, that works for some extensions that have official packages (like jupyterlab-lsp), but not all extensions have packages, but I can probably do a derivation to create them from their github repos. But related issue. Jupyterlab-git needs regular git installed, which it obvioulsy is on the base system, but the jupyterhub/lab environment for some reason can't see git. I must need to load it into the hub/lab environment. They are built with this:

jupyterhubEnv = pkgs.python311.withPackages (p: with p; [
    jupyterhub
    jupyterhub-systemdspawner
    ...

Yes, that works for some extensions that have official packages (like jupyterlab-lsp), but not all extensions have packages, but I can probably do a derivation to create them from their github repos. But related issue. Jupyterlab-git needs regular git installed, which it obvioulsy is on the base system, but the jupyterhub/lab environment for some reason can't see git. I must need to load it into the hub/lab environment. They are built with this:

jupyterhubEnv = pkgs.python311.withPackages (p: with p; [
    jupyterhub
    jupyterhub-systemdspawner
    ...

Is it possible to load non python311pacakges into this enviornment. I tried a few different ways, but couldn't get it to work. Trying to load the regular git nixos package.
Thanks!

Yes, that works for some extensions that have official packages (like jupyterlab-lsp), but not all extensions have packages, but I can probably do a derivation to create them from their github repos. But related issue. Jupyterlab-git needs regular git installed, which it obvioulsy is on the base system, but the jupyterhub/lab environment for some reason can't see git. I must need to load it into the hub/lab environment. They are built like this:

jupyterhubEnv = pkgs.python311.withPackages (p: with p; [
    jupyterhub
    jupyterhub-systemdspawner
    pip
    ...

It appears you can only load python packages into the underlying environment. Is there a way to load non python packages, such as git?
Thanks.

Yes, that works for some extensions that have official packages (like jupyterlab-lsp), but not all extensions have packages, but I can probably do a derivation to create them from their github repos. But related issue. Jupyterlab-git needs regular git installed, which it obviously is on the base system, but the jupyterhub/lab environment for some reason can't see git. I must need to load it into the hub/lab environment. They are built like this:

jupyterhubEnv = pkgs.python311.withPackages (p: with p; [
    jupyterhub
    jupyterhub-systemdspawner
    pip
    ...

It appears you can only load python packages into the underlying environment. Is there a way to load non python packages, such as git?

I'm not sure how you are working. Personally, I use nix-shell to package all the development environment (including python modules and non-python packages). Here is an example for jupyter lab :

let
  nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/04220ed6763637e5899980f98d5c8424b1079353.tar.gz";
  pkgs = import nixpkgs { config = {}; overlays = []; };
in
  pkgs.mkShell {
    packages = with pkgs; [
      (python310.withPackages (ps: with ps; [
        numpy
        pandas
        matplotlib
        scipy
        jupyterlab
        ipympl
        jupyterlab-git
      ]))
      git
    ];

  shellHook = ''
    jupyter lab
  '';
  }

Note that unless you use nix-shell --pure, you don't need to add git if it is alrready present at OS level, although it's probability better to add it anyway for portability

In reply to @benoitdr:matrix.org
Is everyone OK with that ? I guess uploading large datasets to the nix store will increase the nix store hosting costs (on S3 ?). Another option could be to store the datasets in dedicated platforms (HugggingFace, Kaggle, ....) and store pointers in the nix store.