In reply to @pepe:runas.rocks
😭

In reply to @tyberius_prime:matrix.org
I know I build it with python 3.12, bcrypt 4.1.3 and the poetry2nix master like 2 weeks ago.. if you want I can drop you the flake.nix/pyproject.toml/poetry.lock

I tried with this:

                overrides = defaultPoetryOverrides.extend (self: super: {
                  bcrypt = super.bcrypt.overridePythonAttrs (old: {
                    preferWheels = false;
                  });

But it doesn't work :-(

I have a monorepo structured like so:

repo/
    services/
        a/
            pyproject.toml
         b/
            pyproject.toml

Where a imports b as editable with b = {path="../b", develop=true} in the pyproject.toml. I'm having trouble getting my nix development derivation to import b correctly into the shell.

I've tried:

1. a/flake.nix with a derivation like:

        devShells.default = let 
            envShell = mkPoetryEnv {
              projectDir = self;
              overrides = p2n-overrides;
              preferWheels = true;
              editablePackageSources = {
                b = ../b;
              };
            }; 
          in
          pkgs.mkShell {
            name = "A";
            buildInputs = [ envShell ];
            #inputsFrom = [ self.packages.${system}.a ];
            packages = with pkgs; [ poetry docker-compose unzip ruff-lsp stdenv.cc.cc.lib zip awscli2 postgresql];
            LD_LIBRARY_PATH = "${pkgs.stdenv.cc.cc.lib}/lib";
        };
      });

My understanding was that nix (like docker) copies the entire sub-directory structure, so it might be missing "sibling" folders (i.e. w/ same parent directory)
2. I've also tried putting flake.nix in services/ with something like

        devShells.default = let 
            envShell = mkPoetryEnv {
              projectDir = ./a;
              overrides = p2n-overrides;
              preferWheels = true;
              editablePackageSources = {
                b = ./b;
                a = ./a;
              };
            }; 
          in
          pkgs.mkShell {
            name = "A";
            buildInputs = [ envShell ];
            packages = with pkgs; [ poetry docker-compose unzip ruff-lsp stdenv.cc.cc.lib zip awscli2 postgresql];
            LD_LIBRARY_PATH = "${pkgs.stdenv.cc.cc.lib}/lib";
        };
      });

But I still get the ModuleNotFoundError for b.

What's the expected way for me to make this structure work?

In reply to @notwren:matrix.org

I have a monorepo structured like so:

repo/
    services/
        a/
            pyproject.toml
         b/
            pyproject.toml

Where a imports b as editable with b = {path="../b", develop=true} in the pyproject.toml. I'm having trouble getting my nix development derivation to import b correctly into the shell.

I've tried:

1. a/flake.nix with a derivation like:

        devShells.default = let 
            envShell = mkPoetryEnv {
              projectDir = self;
              overrides = p2n-overrides;
              preferWheels = true;
              editablePackageSources = {
                b = ../b;
              };
            }; 
          in
          pkgs.mkShell {
            name = "A";
            buildInputs = [ envShell ];
            #inputsFrom = [ self.packages.${system}.a ];
            packages = with pkgs; [ poetry docker-compose unzip ruff-lsp stdenv.cc.cc.lib zip awscli2 postgresql];
            LD_LIBRARY_PATH = "${pkgs.stdenv.cc.cc.lib}/lib";
        };
      });

My understanding was that nix (like docker) copies the entire sub-directory structure, so it might be missing "sibling" folders (i.e. w/ same parent directory)
2. I've also tried putting flake.nix in services/ with something like

        devShells.default = let 
            envShell = mkPoetryEnv {
              projectDir = ./a;
              overrides = p2n-overrides;
              preferWheels = true;
              editablePackageSources = {
                b = ./b;
                a = ./a;
              };
            }; 
          in
          pkgs.mkShell {
            name = "A";
            buildInputs = [ envShell ];
            packages = with pkgs; [ poetry docker-compose unzip ruff-lsp stdenv.cc.cc.lib zip awscli2 postgresql];
            LD_LIBRARY_PATH = "${pkgs.stdenv.cc.cc.lib}/lib";
        };
      });

But I still get the ModuleNotFoundError for b.

What's the expected way for me to make this structure work?

I have some thing like

        devShells.a = pkgs.mkShell {
          packages = with pkgs; [
            (mkPoetryEnv {
              projectDir = a/.;
              python=pkgs.python39;
            })
          ];
        };

        devShells.b = pkgs.mkShell {
          packages = with pkgs; [
            (mkPoetryEnv {
              projectDir = b/.;
              python=pkgs.python39;
            })
          ];
        };

I'm wondering if poetry2nix could be used to build wheels of all my dependencies?
Currently, I use poetry2nix for a lovely dev environment but now I want to build wheels for my production environment.
I'm using docker currently by exporting my poetry.lock to a requirements.txt and then for each package doing
python wheel <packageName==Version>

I have to maintain dependencies manually in the Docker file :(
Feels like I should be able to do this with nix because it already created a reproducible environment to build those packages.
I just need the wheel output?

Ah yes

ldd M2Crypto/_m2crypto.cpython-39-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffc1013c000)
        libssl.so.3 => /nix/store/gkx5bfa7kf83wgh94s6vr93vn356w081-openssl-3.0.14/lib/libssl.so.3 (0x00007f825e337000)
        libcrypto.so.3 => /nix/store/gkx5bfa7kf83wgh94s6vr93vn356w081-openssl-3.0.14/lib/libcrypto.so.3 (0x00007f825de00000)
        libc.so.6 => /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6 (0x00007f825dc13000)
        libdl.so.2 => /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libdl.so.2 (0x00007f825e332000)
        libpthread.so.0 => /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libpthread.so.0 (0x00007f825e32b000)
        /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib64/ld-linux-x86-64.so.2 (0x00007f825e488000)

Hello,

I am scratching my head against a problem I have.
It might be related to my low level in Nix but anayway I am running out of idea and wanted to ask you for help.

What I am trying to do ?

I want to transform a python application into a Nix package with all the dependencies to either integrate it to a NixOS server, or to just run the application from a system with Nix.

The project has few specific point :

• It contains a module/library with the same name as the project;
• A bin folder exist and contains multiple python files which call Popen to run other scripts located into the bin folder.
• One of the script inside the bin folder runs a Flask application with gunicorn.

What I was able to do ?

I was able to successfully run the project inside a shell with the command nix develop;

I was able to build my packages with nix build.
However I cannot run it because I have this issue :

─○ ./result/bin/start
Traceback (most recent call last):
    File "/nix/store/4h6llg34gcf1gbw4gcxki26ib22ys3h9-python3.12-test-0.0.0/bin/.start-wrapped", line 6, in <module>
        from bin.start import main
    ModuleNotFoundError: No module named 'bin'

The workaround I found was to set the PYTHONPATH variable manually :

─○ TEST_HOME='/home/user/Repositories/TestFlake' PYTHONPATH=$(pwd)/result/:$PYTHONPATH ./result/bin/start
Start backend (redis)...
Waiting on cache to start
Waiting on indexing to start
+ redis-server ./cache.conf
+ redis-server ./indexing.conf
done.
Start website...
done.

However I still face issue in my code where flask was not imported :

│  30   │ File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
│  31   │ File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
│  32   │ File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
│  33   │ File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
│  34   │ File "<frozen importlib._bootstrap_external>", line 995, in exec_module
│  35   │ File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
│  36   │ File "/home/user/Repositories/TestFlake/website/web/__init__.py", line 9, in <module>
│  37   │ │ from flask import (
│  38   ModuleNotFoundError: No module named 'flask'

What flake is use ?

{
  description = "Application packaged using poetry2nix";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    poetry2nix = {
      url = "github:nix-community/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    flake-utils.lib.eachDefaultSystem
      (system:
        let
          # see https://github.com/nix-community/poetry2nix/tree/master#api for more functions and examples.
          pkgs = nixpkgs.legacyPackages.${system};
          inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; }) mkPoetryApplication defaultPoetryOverrides mkPoetryPackages mkPoetryScriptsPackage;

          pypkgs-build-requirements = {
            fsspec = [ "hatchling" "hatch-vcs" ];
            har2tree = [ "poetry" ];
            pydeep2 = [ "setuptools-scm" "ssdeep" pkgs.ssdeep pkgs.libtool pkgs.automake ];
            vt-py = [ "setuptools" "pytest-runner" ];
            pyeupi = [ "poetry" ];
            pyfaup = [ "setuptools" ];
            types-cffi = [ "setuptools" ];
            pypdns = [ "poetry" ];
            passivetotal = [ "setuptools" ];
            pyhashlookup = [ "poetry" ];
            pyipasnhistory = [ "poetry" ];
            pylookyloo = [ "poetry" ];
            pymisp = [ "poetry" ];
            pylookyloomonitoring = [ "poetry" ];
            pypandora = [ "poetry" ];
            pyphishtanklookup = [ "poetry" ];
            pysanejs = [ "poetry" ];
            pysecuritytxt = [ "poetry" ];
            pytaxonomies = [ "poetry" ];
            ssdeep = [ "setuptools" ];
          };

          #p2n-overrides = poetry2nix.overrides.withDefaults (final: prev:
          p2n-overrides = defaultPoetryOverrides.extend
            (final: prev:
              builtins.mapAttrs
                (package: build-requirements:
                  (builtins.getAttr package prev).overridePythonAttrs (old: {
                    buildInputs = (old.buildInputs or [ ]) ++ (builtins.map (pkg: if builtins.isString pkg then builtins.getAttr pkg prev else pkg) build-requirements);
                    #preferWheel = true;
                  })
                )

                pypkgs-build-requirements
            );

          lookylooPackage = mkPoetryPackages
            {
              projectDir = ./.;
              overrides = p2n-overrides;

            };
          lookylooApp = mkPoetryApplication {
            projectDir = ./.;
            overrides = p2n-overrides;

            # patches = [
            #   ./nix/cache.patch
            #   ./nix/indexing.patch
            # ];

            dependencies = [
              pkgs.redis
              pkgs.ffmpeg
            ];

            nativeBuildInputs = [
              pkgs.ssdeep
            ] ++ pkgs.lib.mapAttrsToList
              (name: value:
                pkgs.writers.writePython3Bin "${name}" { flakeIgnore = [ "E401" "E501" ]; } ''
                  import sys, importlib
                  mod, attr = "${value}".split(":", 1)
                  sys.exit(getattr(importlib.import_module(mod), attr)())
                ''
              )
              (builtins.fromTOML (builtins.readFile ./pyproject.toml)).tool.poetry.scripts;
          };

        in
          packages = {
            default = lookylooApp;
          };


          devShells.default =
            pkgs.mkShell
              {
                inputsFrom = [
                  lookylooPackage
                ];

                packages = [
                  lookylooApp
                  #  pkgs.poetry
                  #  pkgs.python3
                ];

                shellHook = ''
                  export LOOKYLOO_HOME="$(pwd)";
                  export PYTHONPATH="$(pwd):$PYTHONPATH"
                  export PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=true

                  echo "[+] Welcome to this nix shell"
                  echo "- Python $(python --version)"
                '';
              };
        });
}

What the problem is from my understanding ?

I think I am facing multiple issues :

• I am not sure to package it well and to do it in a "Nix" way ;
• On the build phase I am not abble to retrieve the module named bin without manually set the PYTHONPATH;
• Even with the PYTHONPATH I still miss dependencies.

I am at a point where I tested everything, even absurd things.

I created a test project to highlight my issues.
It is much lighter than the original project : https://gitea.com/oven_spinout988/TestFlake.

Please let me know if I should post my message elsewhere.

Would someone can help or assit me in the process please ?

Hello,

I am scratching my head against a problem I have.
It might be related to my low level in Nix but anayway I am running out of idea and wanted to ask you for help.

What I am trying to do ?

I want to transform a python application into a Nix package with all the dependencies to either integrate it to a NixOS server, or to just run the application from a system with Nix.

The project has few specific point :

• It contains a module/library with the same name as the project;
• A bin folder exist and contains multiple python files which call Popen to run other scripts located into the bin folder.
• One of the script inside the bin folder runs a Flask application with gunicorn.

What I was able to do ?

I was able to successfully run the project inside a shell with the command nix develop;

I was able to build my packages with nix build.
However I cannot run it because I have this issue :

─○ ./result/bin/start
Traceback (most recent call last):
    File "/nix/store/4h6llg34gcf1gbw4gcxki26ib22ys3h9-python3.12-test-0.0.0/bin/.start-wrapped", line 6, in <module>
        from bin.start import main
    ModuleNotFoundError: No module named 'bin'

The workaround I found was to set the PYTHONPATH variable manually :

─○ TEST_HOME='/home/user/Repositories/TestFlake' PYTHONPATH=$(pwd)/result/:$PYTHONPATH ./result/bin/start
Start backend (redis)...
Waiting on cache to start
Waiting on indexing to start
+ redis-server ./cache.conf
+ redis-server ./indexing.conf
done.
Start website...
done.

However I still face issue in my code where flask was not imported :

│  30   │ File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
│  31   │ File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
│  32   │ File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
│  33   │ File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
│  34   │ File "<frozen importlib._bootstrap_external>", line 995, in exec_module
│  35   │ File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
│  36   │ File "/home/user/Repositories/TestFlake/website/web/__init__.py", line 9, in <module>
│  37   │ │ from flask import (
│  38   ModuleNotFoundError: No module named 'flask'

What flake is use ?

{
  description = "Application packaged using poetry2nix";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    poetry2nix = {
      url = "github:nix-community/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    flake-utils.lib.eachDefaultSystem
      (system:
        let
          # see https://github.com/nix-community/poetry2nix/tree/master#api for more functions and examples.
          pkgs = nixpkgs.legacyPackages.${system};
          inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; }) mkPoetryApplication defaultPoetryOverrides mkPoetryPackages mkPoetryScriptsPackage;

          pypkgs-build-requirements = {
            fsspec = [ "hatchling" "hatch-vcs" ];
            har2tree = [ "poetry" ];
            pydeep2 = [ "setuptools-scm" "ssdeep" pkgs.ssdeep pkgs.libtool pkgs.automake ];
            vt-py = [ "setuptools" "pytest-runner" ];
            pyeupi = [ "poetry" ];
            pyfaup = [ "setuptools" ];
            types-cffi = [ "setuptools" ];
            pypdns = [ "poetry" ];
            passivetotal = [ "setuptools" ];
            pyhashlookup = [ "poetry" ];
            pyipasnhistory = [ "poetry" ];
            pylookyloo = [ "poetry" ];
            pymisp = [ "poetry" ];
            pylookyloomonitoring = [ "poetry" ];
            pypandora = [ "poetry" ];
            pyphishtanklookup = [ "poetry" ];
            pysanejs = [ "poetry" ];
            pysecuritytxt = [ "poetry" ];
            pytaxonomies = [ "poetry" ];
            ssdeep = [ "setuptools" ];
          };

          #p2n-overrides = poetry2nix.overrides.withDefaults (final: prev:
          p2n-overrides = defaultPoetryOverrides.extend
            (final: prev:
              builtins.mapAttrs
                (package: build-requirements:
                  (builtins.getAttr package prev).overridePythonAttrs (old: {
                    buildInputs = (old.buildInputs or [ ]) ++ (builtins.map (pkg: if builtins.isString pkg then builtins.getAttr pkg prev else pkg) build-requirements);
                    #preferWheel = true;
                  })
                )

                pypkgs-build-requirements
            );

          lookylooPackage = mkPoetryPackages
            {
              projectDir = ./.;
              overrides = p2n-overrides;

            };
          lookylooApp = mkPoetryApplication {
            projectDir = ./.;
            overrides = p2n-overrides;

            # patches = [
            #   ./nix/cache.patch
            #   ./nix/indexing.patch
            # ];

            dependencies = [
              pkgs.redis
              pkgs.ffmpeg
            ];

            nativeBuildInputs = [
              pkgs.ssdeep
            ] ++ pkgs.lib.mapAttrsToList
              (name: value:
                pkgs.writers.writePython3Bin "${name}" { flakeIgnore = [ "E401" "E501" ]; } ''
                  import sys, importlib
                  mod, attr = "${value}".split(":", 1)
                  sys.exit(getattr(importlib.import_module(mod), attr)())
                ''
              )
              (builtins.fromTOML (builtins.readFile ./pyproject.toml)).tool.poetry.scripts;
          };

        in
          packages = {
            default = lookylooApp;
          };


          devShells.default =
            pkgs.mkShell
              {
                inputsFrom = [
                  lookylooPackage
                ];

                packages = [
                  lookylooApp
                  #  pkgs.poetry
                  #  pkgs.python3
                ];

                shellHook = ''
                  export LOOKYLOO_HOME="$(pwd)";
                  export PYTHONPATH="$(pwd):$PYTHONPATH"
                  export PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=true

                  echo "[+] Welcome to this nix shell"
                  echo "- Python $(python --version)"
                '';
              };
        });
}

What the problem is from my understanding ?

I think I am facing multiple issues :

• I am not sure to package it well and to do it in a "Nix" way ;
• On the build phase I am not abble to retrieve the module named bin without manually set the PYTHONPATH;
• Even with the PYTHONPATH I still miss dependencies.

I am at a point where I tested everything, even absurd things.

I created a test project to highlight my issues.
It is much lighter than the original project : https://gitea.com/oven_spinout988/TestFlake.

Please let me know if I should post my message elsewhere.

Would someone can help or assit me in the process please ?