In reply to @benoitdr:matrix.org
Still a question : I have successfully packaged ctransformers based on the information above, but it cannot see my GPU. If I would pip install it, I would use the cuda option (pip install ctransfomers[cuda]). Is there a way to pass the same option to buildPythonPackage ?

(partially) answering to myslef, looking at setup.py, I can see that the cuda option corresponds to 2 extra packages :

        "cuda": [
            "nvidia-cuda-runtime-cu12",
            "nvidia-cublas-cu12",
        ],

In reply to @benoitdr:matrix.org
Still a question : I have successfully packaged ctransformers based on the information above, but it cannot see my GPU. If I would pip install it, I would use the cuda option (pip install ctransfomers[cuda]). Is there a way to pass the same option to buildPythonPackage ?

(casual reminder that these are poetry2nix-generated wrappers for the pypi wheels, expect them to break)

EDIT: aj this is from setup.py, nvm

cudaPackages.cudatoolkit

You can remove this one

python310Packages.pycuda

I don't see pycuda in ctrasnformers' dependencies?

In reply to @ss:someonex.net
You need to pass this flag during the ctransformers build: https://github.com/marella/ctransformers/blob/ed02cf4b9322435972ff3566fd4832806338ca3d/CMakeLists.txt#L6

Still I need cudatoolkit in propagatedBuildInputs

cudaPackages.cudatoolkit is being deprecated, you do not need it; cf. nixpkgs' torch or opencv4 derivations for examples

In reply to @benoitdr:matrix.org
Strange ... It doesn't work without it on my side

Here is my shell.nix file :

let
  nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/04220ed6763637e5899980f98d5c8424b1079353.tar.gz";
  pkgs = import nixpkgs { config = { allowUnfree = true; }; overlays = []; };
  ctransformers = pkgs.python310.pkgs.buildPythonPackage rec {
      pname = "ctransformers";
      version = "0.2.27";
      format = "setuptools";
      src = pkgs.python310.pkgs.fetchPypi {
        inherit pname version;
        sha256 = "25653d4be8a5ed4e2d3756544c1e9881bf95404be5371c3ed506a256c28663d5";
      };
      doCheck = false;
      dontUseCmakeConfigure = true;
      nativeBuildInputs = with pkgs; [
        python310Packages.setuptools
        python310Packages.scikit-build
        python310Packages.ninja
        python310Packages.cmake
      ];
      propagatedBuildInputs = with pkgs; [
        python310Packages.huggingface-hub
        python310Packages.py-cpuinfo
        cudaPackages.cudatoolkit
      ];
      env.CT_CUBLAS = "ON";
    };
in
  pkgs.mkShell {
    packages = with pkgs; [
      (python310.withPackages (ps: with ps; [
        ctransformers
      ]))
    ];
  }

Without cudatoolkit, at compile time, ctransformers complaints that cublas is not found, and at runtime, ctransformers cannot use the GPU.
As far as I undertstand, it seems a bit logical to me. How would ctransfomers find the cuda libraries without cudatoolkit ?

BTW, the same thing happens if I set cudaSupport to true like this :

pkgs = import nixpkgs { config = { allowUnfree = true; cudaSupport = true; }; overlays = []; };

cudatoolkit is still needed

In reply to @benoitdr:matrix.org

Here is my shell.nix file :

let
  nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/04220ed6763637e5899980f98d5c8424b1079353.tar.gz";
  pkgs = import nixpkgs { config = { allowUnfree = true; }; overlays = []; };
  ctransformers = pkgs.python310.pkgs.buildPythonPackage rec {
      pname = "ctransformers";
      version = "0.2.27";
      format = "setuptools";
      src = pkgs.python310.pkgs.fetchPypi {
        inherit pname version;
        sha256 = "25653d4be8a5ed4e2d3756544c1e9881bf95404be5371c3ed506a256c28663d5";
      };
      doCheck = false;
      dontUseCmakeConfigure = true;
      nativeBuildInputs = with pkgs; [
        python310Packages.setuptools
        python310Packages.scikit-build
        python310Packages.ninja
        python310Packages.cmake
      ];
      propagatedBuildInputs = with pkgs; [
        python310Packages.huggingface-hub
        python310Packages.py-cpuinfo
        cudaPackages.cudatoolkit
      ];
      env.CT_CUBLAS = "ON";
    };
in
  pkgs.mkShell {
    packages = with pkgs; [
      (python310.withPackages (ps: with ps; [
        ctransformers
      ]))
    ];
  }

Without cudatoolkit, at compile time, ctransformers complaints that cublas is not found, and at runtime, ctransformers cannot use the GPU.
As far as I undertstand, it seems a bit logical to me. How would ctransfomers find the cuda libraries without cudatoolkit ?

By you llisting individual cuda libraries and nvcc as inputs, cf torch/default.nix and opencv/4.nix for the examples

The documentation needs to be clarified about this...

Indeed a bit of documentation would help ;-)
Looking at cudatoolkit sources, I can see that it's importing many many things I don't need, so I would be happy to get rid of it.
Unfortunately, if I replace it by cuda_nvcc, cuda_cudart and libcublas, ctransformers doesn't build anymore.

-- Using CUDA architectures: 52;61;70
-- Unable to find cuda_runtime.h in "/nix/store/p8058x6fpdlw7hy72qsqn41qhllqncgm-cuda_nvcc-11.8.89/include" for CUDAToolkit_INCLUDE_DIR.
-- Unable to find cublas_v2.h in either "" or "/nix/math_libs/include"
-- Could NOT find CUDAToolkit (missing: CUDAToolkit_INCLUDE_DIR) (found version "11.8.89")
CMake Warning at CMakeLists.txt:163 (message):
  cuBLAS not found

Looking at CMakeLists.txt :

if (CT_CUBLAS)
    find_package(CUDAToolkit)

So it seems there is a problem with the CUDAToolkit_INCLUDE_DIR env variable.
Not sure it's related, but looking at https://cmake.org/cmake/help/latest/module/FindCUDAToolkit.html, that function is setting a CUDAToolkit_INCLUDE_DIRS (with extra S)