Hi all, here is my shell.nix for jupyter lab environment (I'm not a flake user so I keep it as shell.nix for the moment).
I'm now trying to add ctransformers library but the build fails miserability (see the log, problem to locate setup.py as I understand, although I see it at the project root). Would anyone have an idea ?

let
  nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/04220ed6763637e5899980f98d5c8424b1079353.tar.gz";
  pkgs = import nixpkgs { config = {}; overlays = []; };
  ctransformers = pkgs.python310.pkgs.buildPythonPackage rec {
      pname = "ctransformers";
      version = "0.2.27";
      format = "pyproject";
      src = pkgs.python310.pkgs.fetchPypi {
        inherit pname version;
        sha256 = "25653d4be8a5ed4e2d3756544c1e9881bf95404be5371c3ed506a256c28663d5";
      };
      doCheck = false;
      nativeBuildInputs = [
        pkgs.python310Packages.setuptools
        pkgs.python310Packages.scikit-build
        pkgs.python310Packages.ninja
        pkgs.python310Packages.cmake
        pkgs.python310Packages.huggingface-hub
        pkgs.python310Packages.py-cpuinfo
        pkgs.python310Packages.huggingface-hub
        pkgs.python310Packages.py-cpuinfo
      ];
    };
in
  pkgs.mkShell {
    packages = with pkgs; [
      (python310.withPackages (ps: with ps; [
        numpy
        pandas
        matplotlib
        scipy
        jupyterlab
        ipympl
        jupyterlab-git
        ctransformers
      ]))
    ];

  shellHook = ''
    jupyter lab
    exit
  '';
  }

and here is the error message :

-- Build files have been written to: /build/ctransformers-0.2.27/build
cmake: enabled parallel building
cmake: enabled parallel installing
building
Executing pypaBuildPhase
Creating a wheel...
ERROR Source /build/ctransformers-0.2.27/build does not appear to be a Python project: no pyproject.toml or setup.py
error: builder for '/nix/store/cwk0kjniclqcskv3i5gxfw5nihld0md9-python3.10-ctransformers-0.2.27.drv' failed with exit code 1;
       last 10 log lines:
       >     CMAKE_POLICY_DEFAULT_CMP0025
       >
       > 
       > -- Build files have been written to: /build/ctransformers-0.2.27/build
       > cmake: enabled parallel building
       > cmake: enabled parallel installing
       > building
       > Executing pypaBuildPhase
       > Creating a wheel...
       > ERROR Source /build/ctransformers-0.2.27/build does not appear to be a Python project: no pyproject.toml or setup.py
       For full logs, run 'nix log /nix/store/cwk0kjniclqcskv3i5gxfw5nihld0md9-python3.10-ctransformers-0.2.27.drv'.
error: 1 dependencies of derivation '/nix/store/r07h95i2csyx80f0lsf05v7i4jciqxg2-python3-3.10.13-env.drv' failed to build

In reply to @benoitdr:matrix.org

Hi all, here is my shell.nix for jupyter lab environment (I'm not a flake user so I keep it as shell.nix for the moment).
I'm now trying to add ctransformers library but the build fails miserability (see the log, problem to locate setup.py as I understand, although I see it at the project root). Would anyone have an idea ?

let
  nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/04220ed6763637e5899980f98d5c8424b1079353.tar.gz";
  pkgs = import nixpkgs { config = {}; overlays = []; };
  ctransformers = pkgs.python310.pkgs.buildPythonPackage rec {
      pname = "ctransformers";
      version = "0.2.27";
      format = "pyproject";
      src = pkgs.python310.pkgs.fetchPypi {
        inherit pname version;
        sha256 = "25653d4be8a5ed4e2d3756544c1e9881bf95404be5371c3ed506a256c28663d5";
      };
      doCheck = false;
      nativeBuildInputs = [
        pkgs.python310Packages.setuptools
        pkgs.python310Packages.scikit-build
        pkgs.python310Packages.ninja
        pkgs.python310Packages.cmake
        pkgs.python310Packages.huggingface-hub
        pkgs.python310Packages.py-cpuinfo
        pkgs.python310Packages.huggingface-hub
        pkgs.python310Packages.py-cpuinfo
      ];
    };
in
  pkgs.mkShell {
    packages = with pkgs; [
      (python310.withPackages (ps: with ps; [
        numpy
        pandas
        matplotlib
        scipy
        jupyterlab
        ipympl
        jupyterlab-git
        ctransformers
      ]))
    ];

  shellHook = ''
    jupyter lab
    exit
  '';
  }

and here is the error message :

-- Build files have been written to: /build/ctransformers-0.2.27/build
cmake: enabled parallel building
cmake: enabled parallel installing
building
Executing pypaBuildPhase
Creating a wheel...
ERROR Source /build/ctransformers-0.2.27/build does not appear to be a Python project: no pyproject.toml or setup.py
error: builder for '/nix/store/cwk0kjniclqcskv3i5gxfw5nihld0md9-python3.10-ctransformers-0.2.27.drv' failed with exit code 1;
       last 10 log lines:
       >     CMAKE_POLICY_DEFAULT_CMP0025
       >
       > 
       > -- Build files have been written to: /build/ctransformers-0.2.27/build
       > cmake: enabled parallel building
       > cmake: enabled parallel installing
       > building
       > Executing pypaBuildPhase
       > Creating a wheel...
       > ERROR Source /build/ctransformers-0.2.27/build does not appear to be a Python project: no pyproject.toml or setup.py
       For full logs, run 'nix log /nix/store/cwk0kjniclqcskv3i5gxfw5nihld0md9-python3.10-ctransformers-0.2.27.drv'.
error: 1 dependencies of derivation '/nix/store/r07h95i2csyx80f0lsf05v7i4jciqxg2-python3-3.10.13-env.drv' failed to build

In reply to @benoitdr:matrix.org
Thanks, dontUseCmakeConfigure = true was the missing piece

In reply to @trexd:matrix.org
Yeah protip, if you're looking for AI packages, use github search and do language:Nix PACKAGE_NAME buildPythonPackage and there's a good chance someone has already packaged it. That saves me so much time :D

In reply to @benoitdr:matrix.org
Still a question : I have successfully packaged ctransformers based on the information above, but it cannot see my GPU. If I would pip install it, I would use the cuda option (pip install ctransfomers[cuda]). Is there a way to pass the same option to buildPythonPackage ?

(partially) answering to myslef, looking at setup.py, I can see that the cuda option corresponds to 2 extra packages :

        "cuda": [
            "nvidia-cuda-runtime-cu12",
            "nvidia-cublas-cu12",
        ],

In reply to @benoitdr:matrix.org
Still a question : I have successfully packaged ctransformers based on the information above, but it cannot see my GPU. If I would pip install it, I would use the cuda option (pip install ctransfomers[cuda]). Is there a way to pass the same option to buildPythonPackage ?

(casual reminder that these are poetry2nix-generated wrappers for the pypi wheels, expect them to break)

EDIT: aj this is from setup.py, nvm

cudaPackages.cudatoolkit

You can remove this one

python310Packages.pycuda

I don't see pycuda in ctrasnformers' dependencies?

In reply to @ss:someonex.net
You need to pass this flag during the ctransformers build: https://github.com/marella/ctransformers/blob/ed02cf4b9322435972ff3566fd4832806338ca3d/CMakeLists.txt#L6

Still I need cudatoolkit in propagatedBuildInputs

cudaPackages.cudatoolkit is being deprecated, you do not need it; cf. nixpkgs' torch or opencv4 derivations for examples