Hello,

I am scratching my head against a problem I have.
It might be related to my low level in Nix but anayway I am running out of idea and wanted to ask you for help.

What I am trying to do ?

I want to transform a python application into a Nix package with all the dependencies to either integrate it to a NixOS server, or to just run the application from a system with Nix.

The project has few specific point :

• It contains a module/library with the same name as the project;
• A bin folder exist and contains multiple python files which call Popen to run other scripts located into the bin folder.
• One of the script inside the bin folder runs a Flask application with gunicorn.

What I was able to do ?

I was able to successfully run the project inside a shell with the command nix develop;

I was able to build my packages with nix build.
However I cannot run it because I have this issue :

─○ ./result/bin/start
Traceback (most recent call last):
    File "/nix/store/4h6llg34gcf1gbw4gcxki26ib22ys3h9-python3.12-test-0.0.0/bin/.start-wrapped", line 6, in <module>
        from bin.start import main
    ModuleNotFoundError: No module named 'bin'

The workaround I found was to set the PYTHONPATH variable manually :

─○ TEST_HOME='/home/user/Repositories/TestFlake' PYTHONPATH=$(pwd)/result/:$PYTHONPATH ./result/bin/start
Start backend (redis)...
Waiting on cache to start
Waiting on indexing to start
+ redis-server ./cache.conf
+ redis-server ./indexing.conf
done.
Start website...
done.

However I still face issue in my code where flask was not imported :

│  30   │ File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
│  31   │ File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
│  32   │ File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
│  33   │ File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
│  34   │ File "<frozen importlib._bootstrap_external>", line 995, in exec_module
│  35   │ File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
│  36   │ File "/home/user/Repositories/TestFlake/website/web/__init__.py", line 9, in <module>
│  37   │ │ from flask import (
│  38   ModuleNotFoundError: No module named 'flask'

What flake is use ?

https://gitea.com/oven_spinout988/TestFlake/src/branch/main/flake.nix

What the problem is from my understanding ?

I think I am facing multiple issues :

• I am not sure to package it well and to do it in a "Nix" way ;
• On the build phase I am not abble to retrieve the module named bin without manually set the PYTHONPATH;
• Even with the PYTHONPATH I still miss dependencies.

I am at a point where I tested everything, even absurd things.

I created a test project to highlight my issues.
It is much lighter than the original project : https://gitea.com/oven_spinout988/TestFlake.

Please let me know if I should post my message elsewhere.

Would someone can help or assit me in the process please ?

Hello,

I am scratching my head against a problem I have.
It might be related to my low level in Nix but anayway I am running out of idea and wanted to ask you for help.

What I am trying to do ?

I want to transform a python application into a Nix package with all the dependencies to either integrate it to a NixOS server, or to just run the application from a system with Nix.

The project has few specific point :

• It contains a module/library with the same name as the project;
• A bin folder exist and contains multiple python files which call Popen to run other scripts located into the bin folder.
• One of the script inside the bin folder runs a Flask application with gunicorn.

What I was able to do ?

I was able to successfully run the project inside a shell with the command nix develop;

I was able to build my packages with nix build.
However I cannot run it because I have this issue :

─○ ./result/bin/start
Traceback (most recent call last):
    File "/nix/store/4h6llg34gcf1gbw4gcxki26ib22ys3h9-python3.12-test-0.0.0/bin/.start-wrapped", line 6, in <module>
        from bin.start import main
    ModuleNotFoundError: No module named 'bin'

The workaround I found was to set the PYTHONPATH variable manually :

─○ TEST_HOME='/home/user/Repositories/TestFlake' PYTHONPATH=$(pwd)/result/:$PYTHONPATH ./result/bin/start
Start backend (redis)...
Waiting on cache to start
Waiting on indexing to start
+ redis-server ./cache.conf
+ redis-server ./indexing.conf
done.
Start website...
done.

However I still face issue in my code where flask was not imported :

│  30   │ File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
│  31   │ File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
│  32   │ File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
│  33   │ File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
│  34   │ File "<frozen importlib._bootstrap_external>", line 995, in exec_module
│  35   │ File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
│  36   │ File "/home/user/Repositories/TestFlake/website/web/__init__.py", line 9, in <module>
│  37   │ │ from flask import (
│  38   ModuleNotFoundError: No module named 'flask'

What flake is use ?

https://gitea.com/oven_spinout988/TestFlake/src/branch/main/flake.nix

What the problem is from my understanding ?

I think I am facing multiple issues :

• I am not sure to package it well and to do it in a "Nix" way ;
• On the build phase I am not abble to retrieve the module named bin without manually set the PYTHONPATH;
• Even with the PYTHONPATH I still miss dependencies.

I am at a point where I tested everything, even absurd things.

I created a test project to highlight my issues.
It is much lighter than the original project : https://gitea.com/oven_spinout988/TestFlake.

Please let me know if I should post my message elsewhere.

Would someone can help or assit me in the process please ?

Hello,

I am scratching my head against a problem I have.
It might be related to my low level in Nix but anayway I am running out of idea and wanted to ask you for help.

What I am trying to do ?

I want to transform a python application into a Nix package with all the dependencies to either integrate it to a NixOS server, or to just run the application from a system with Nix.

The project has few specific point :

• It contains a module/library with the same name as the project;
• A bin folder exist and contains multiple python files which call Popen to run other scripts located into the bin folder.
• One of the script inside the bin folder runs a Flask application with gunicorn.

What I was able to do ?

I was able to successfully run the project inside a shell with the command nix develop;

I was able to build my packages with nix build.
However I cannot run it because I have this issue :

─○ ./result/bin/start
Traceback (most recent call last):
    File "/nix/store/4h6llg34gcf1gbw4gcxki26ib22ys3h9-python3.12-test-0.0.0/bin/.start-wrapped", line 6, in <module>
        from bin.start import main
    ModuleNotFoundError: No module named 'bin'

The workaround I found was to set the PYTHONPATH variable manually :

─○ TEST_HOME='/home/user/Repositories/TestFlake' PYTHONPATH=$(pwd)/result/:$PYTHONPATH ./result/bin/start
Start backend (redis)...
Waiting on cache to start
Waiting on indexing to start
+ redis-server ./cache.conf
+ redis-server ./indexing.conf
done.
Start website...
done.

However I still face issue in my code where flask was not imported :

│  30   │ File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
│  31   │ File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
│  32   │ File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
│  33   │ File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
│  34   │ File "<frozen importlib._bootstrap_external>", line 995, in exec_module
│  35   │ File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
│  36   │ File "/home/user/Repositories/TestFlake/website/web/__init__.py", line 9, in <module>
│  37   │ │ from flask import (
│  38   ModuleNotFoundError: No module named 'flask'

What flake is use ?

https://gitea.com/oven_spinout988/TestFlake/src/branch/main/flake.nix

{
  description = "Application packaged using poetry2nix";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    poetry2nix = {
      url = "github:nix-community/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    flake-utils.lib.eachDefaultSystem
      (system:
        let
          # see https://github.com/nix-community/poetry2nix/tree/master#api for more functions and examples.
          pkgs = nixpkgs.legacyPackages.${system};
          inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; }) mkPoetryApplication defaultPoetryOverrides mkPoetryPackages mkPoetryScriptsPackage;

          pypkgs-build-requirements = {
            types-cffi = [ "setuptools" ];
          };

          #p2n-overrides = poetry2nix.overrides.withDefaults (final: prev:
          p2n-overrides = defaultPoetryOverrides.extend
            (final: prev:
              builtins.mapAttrs
                (package: build-requirements:
                  (builtins.getAttr package prev).overridePythonAttrs (old: {
                    buildInputs = (old.buildInputs or [ ]) ++ (builtins.map (pkg: if builtins.isString pkg then builtins.getAttr pkg prev else pkg) build-requirements);
                    #preferWheel = true;
                  })
                )
                pypkgs-build-requirements
            );

          testPackage = mkPoetryPackages {
            projectDir = ./.;
            overrides = p2n-overrides;
          };

          testApp = mkPoetryApplication {
            projectDir = ./.;
            overrides = p2n-overrides;

            # patches = [
            #   ./nix/cache.patch
            #   ./nix/indexing.patch
            # ];

            dependencies = [
              pkgs.redis
              pkgs.ffmpeg
            ];

            buildInputs = [
              pkgs.python3
              testPackage.poetryPackages
            ] ++ pkgs.lib.mapAttrsToList
              (name: value:
                pkgs.writers.writePython3Bin "${name}" { flakeIgnore = [ "E401" "E501" ]; } ''
                  import sys, importlib
                  mod, attr = "${value}".split(":", 1)
                  sys.exit(getattr(importlib.import_module(mod), attr)())
                ''
              )
              (builtins.fromTOML (builtins.readFile ./pyproject.toml)).tool.poetry.scripts;

            nativeBuildInputs = [
              pkgs.ssdeep
            ];

            # 
            # # --prefix PYTHONPATH : "$PYTHONPATH" \
            # postFixup = ''
            #   wrapProgram "$out/bin/start" \
            #     --prefix PYTHONPATH : "$out:$PYTHONPATH" \
            #     --prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.redis pkgs.ffmpeg ]}
            # '';

            # pkgs.lib.mapAttrsToList
            #     (name: value:
            #       pkgs.writers.writePython3Bin name { flakeIgnore = [ "E401" "E501" ]; } ''
            #         import sys, importlib
            #         mod, attr = "${value}".split(":", 1)
            #         sys.exit(getattr(importlib.import_module(mod), attr)())
            #       ''
            #     )
            #     (builtins.fromTOML (builtins.readFile ./pyproject.toml)).tool.poetry.scripts;
          };

        in
        {
          packages = {
            default = testApp;
          };

          devShells.default =
            pkgs.mkShell
              {
                inputsFrom = [
                  testApp
                ];

                packages = [
                  testApp
                ];

                shellHook = ''
                  export PYTHONPATH="$(pwd):$PYTHONPATH"
                  export TEST_HOME="$(pwd)"

                  echo "[+} Nix shell information"
                  echo "- Python $(python --version)"
                '';
              };
        });
}

**What the problem is from my understanding ?**

I think I am facing multiple issues :

- I am not sure to package it well and to do it in a "Nix" way ;
- On the build phase I am not abble to retrieve the module named `bin` without manually set the `PYTHONPATH`;
- Even with the `PYTHONPATH` I still miss dependencies.

I am at a point where I tested everything, even absurd things.

I created a test project to highlight my issues.
It is much lighter than the original project : https://gitea.com/oven\_spinout988/TestFlake.

Please let me know if I should post my message elsewhere.

Would someone can help or assit me in the process please ?

Hello,

I am scratching my head against a problem I have.
It might be related to my low level in Nix but anayway I am running out of idea and wanted to ask you for help.

What I am trying to do ?

I want to transform a python application into a Nix package with all the dependencies to either integrate it to a NixOS server, or to just run the application from a system with Nix.

The project has few specific point :

• It contains a module/library with the same name as the project;
• A bin folder exist and contains multiple python files which call Popen to run other scripts located into the bin folder.
• One of the script inside the bin folder runs a Flask application with gunicorn.

What I was able to do ?

I was able to successfully run the project inside a shell with the command nix develop;

I was able to build my packages with nix build.
However I cannot run it because I have this issue :

─○ ./result/bin/start
Traceback (most recent call last):
    File "/nix/store/4h6llg34gcf1gbw4gcxki26ib22ys3h9-python3.12-test-0.0.0/bin/.start-wrapped", line 6, in <module>
        from bin.start import main
    ModuleNotFoundError: No module named 'bin'

The workaround I found was to set the PYTHONPATH variable manually :

─○ TEST_HOME='/home/user/Repositories/TestFlake' PYTHONPATH=$(pwd)/result/:$PYTHONPATH ./result/bin/start
Start backend (redis)...
Waiting on cache to start
Waiting on indexing to start
+ redis-server ./cache.conf
+ redis-server ./indexing.conf
done.
Start website...
done.

However I still face issue in my code where flask was not imported :

│  30   │ File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
│  31   │ File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
│  32   │ File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
│  33   │ File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
│  34   │ File "<frozen importlib._bootstrap_external>", line 995, in exec_module
│  35   │ File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
│  36   │ File "/home/user/Repositories/TestFlake/website/web/__init__.py", line 9, in <module>
│  37   │ │ from flask import (
│  38   ModuleNotFoundError: No module named 'flask'

What flake is use ?

https://gitea.com/oven_spinout988/TestFlake/src/branch/main/flake.nix

{
  description = "Application packaged using poetry2nix";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    poetry2nix = {
      url = "github:nix-community/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    flake-utils.lib.eachDefaultSystem
      (system:
        let
          # see https://github.com/nix-community/poetry2nix/tree/master#api for more functions and examples.
          pkgs = nixpkgs.legacyPackages.${system};
          inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; }) mkPoetryApplication defaultPoetryOverrides mkPoetryPackages mkPoetryScriptsPackage;

          pypkgs-build-requirements = {
            types-cffi = [ "setuptools" ];
          };

          #p2n-overrides = poetry2nix.overrides.withDefaults (final: prev:
          p2n-overrides = defaultPoetryOverrides.extend
            (final: prev:
              builtins.mapAttrs
                (package: build-requirements:
                  (builtins.getAttr package prev).overridePythonAttrs (old: {
                    buildInputs = (old.buildInputs or [ ]) ++ (builtins.map (pkg: if builtins.isString pkg then builtins.getAttr pkg prev else pkg) build-requirements);
                    #preferWheel = true;
                  })
                )
                pypkgs-build-requirements
            );

          testPackage = mkPoetryPackages {
            projectDir = ./.;
            overrides = p2n-overrides;
          };

          testApp = mkPoetryApplication {
            projectDir = ./.;
            overrides = p2n-overrides;

            # patches = [
            #   ./nix/cache.patch
            #   ./nix/indexing.patch
            # ];

            dependencies = [
              pkgs.redis
              pkgs.ffmpeg
            ];

            buildInputs = [
              pkgs.python3
              testPackage.poetryPackages
            ] ++ pkgs.lib.mapAttrsToList
              (name: value:
                pkgs.writers.writePython3Bin "${name}" { flakeIgnore = [ "E401" "E501" ]; } ''
                  import sys, importlib
                  mod, attr = "${value}".split(":", 1)
                  sys.exit(getattr(importlib.import_module(mod), attr)())
                ''
              )
              (builtins.fromTOML (builtins.readFile ./pyproject.toml)).tool.poetry.scripts;

            nativeBuildInputs = [
              pkgs.ssdeep
            ];

            # 
            # # --prefix PYTHONPATH : "$PYTHONPATH" \
            # postFixup = ''
            #   wrapProgram "$out/bin/start" \
            #     --prefix PYTHONPATH : "$out:$PYTHONPATH" \
            #     --prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.redis pkgs.ffmpeg ]}
            # '';

            # pkgs.lib.mapAttrsToList
            #     (name: value:
            #       pkgs.writers.writePython3Bin name { flakeIgnore = [ "E401" "E501" ]; } ''
            #         import sys, importlib
            #         mod, attr = "${value}".split(":", 1)
            #         sys.exit(getattr(importlib.import_module(mod), attr)())
            #       ''
            #     )
            #     (builtins.fromTOML (builtins.readFile ./pyproject.toml)).tool.poetry.scripts;
          };

        in
        {
          packages = {
            default = testApp;
          };

          devShells.default =
            pkgs.mkShell
              {
                inputsFrom = [
                  testApp
                ];

                packages = [
                  testApp
                ];

                shellHook = ''
                  export PYTHONPATH="$(pwd):$PYTHONPATH"
                  export TEST_HOME="$(pwd)"

                  echo "[+} Nix shell information"
                  echo "- Python $(python --version)"
                '';
              };
        });
}

What the problem is from my understanding ?

I think I am facing multiple issues :

• I am not sure to package it well and to do it in a "Nix" way ;
• On the build phase I am not abble to retrieve the module named bin without manually set the PYTHONPATH;
• Even with the PYTHONPATH I still miss dependencies.

I am at a point where I tested everything, even absurd things.

I created a test project to highlight my issues.
It is much lighter than the original project : https://gitea.com/oven_spinout988/TestFlake.

Please let me know if I should post my message elsewhere.

Would someone can help or assit me in the process please ?

In reply to @truh:matrix.org
maybe you need to add [tool.poetry.packages] to your pyproject.toml

In reply to @oven_spinout988:matrix.org
I will give it a try

I am not sure if it is a lead or not.
Changing a part of my pyproject.toml made some changes and it is not yelling anymore about the bin module. It is yelling about an other :

[tool.poetry]
name = "test"
version = "0.0.0"
description = "Test."
authors = ["x"]
license = "BSD-3-Clause"
repository = "https://github.com/"
homepage = "https://example.com"
documentation = "https://example.com/docs/main/"

readme = "README.md"

classifiers = [
  'License :: OSI Approved :: BSD License',
  'Operating System :: POSIX :: Linux',
]

packages = [
  { include = "test" },
  { include = "bin" },
  { include = "test" },
  { include = "default", from = "test" },
]
 [..]

Produces :

Traceback (most recent call last):
  File "/nix/store/8835sd6kck0f1ljzvlpp2igzb87v9dnr-python3.12-test-0.0.0/bin/.start-wrapped", line 6, in <module>
    from bin.start import main
  File "/nix/store/8835sd6kck0f1ljzvlpp2igzb87v9dnr-python3.12-test-0.0.0/lib/python3.12/site-packages/bin/start.py", line 5, in <module>
    from test.default import get_homedir
ModuleNotFoundError: No module named 'test.default'

The application is working fine with debian. From the documentation of poetry, it looks like specifying packages disable the auto-detection feature.

In reply to @oven_spinout988:matrix.org
I will give it a try

I am not sure if it is a lead or not.
Changing a part of my pyproject.toml made some changes and it is not yelling anymore about the bin module. It is yelling about an other :

[tool.poetry]
name = "test"
version = "0.0.0"
description = "Test."
authors = ["x"]
license = "BSD-3-Clause"
repository = "https://github.com/"
homepage = "https://example.com"
documentation = "https://example.com/docs/main/"

readme = "README.md"

classifiers = [
  'License :: OSI Approved :: BSD License',
  'Operating System :: POSIX :: Linux',
]

packages = [
  { include = "test" },
  { include = "bin" },
  { include = "default", from = "test" },
]
 [..]

Produces :

Traceback (most recent call last):
  File "/nix/store/8835sd6kck0f1ljzvlpp2igzb87v9dnr-python3.12-test-0.0.0/bin/.start-wrapped", line 6, in <module>
    from bin.start import main
  File "/nix/store/8835sd6kck0f1ljzvlpp2igzb87v9dnr-python3.12-test-0.0.0/lib/python3.12/site-packages/bin/start.py", line 5, in <module>
    from test.default import get_homedir
ModuleNotFoundError: No module named 'test.default'

The application is working fine with debian. From the documentation of poetry, it looks like specifying packages disable the auto-detection feature.

In reply to @truh:matrix.org
oh, test is not a folder with unit tests but application code?

      raise FileNotFoundError(msg)
  FileNotFoundError: Forced include not found: /build/cirrocumulus-1.1.57/build
  error: subprocess-exited-with-error```

why do you try me so?

      raise FileNotFoundError(msg)
  FileNotFoundError: Forced include not found: /build/cirrocumulus-1.1.57/build
  error: subprocess-exited-with-error
```
why do you try me so?

      raise FileNotFoundError(msg)
  FileNotFoundError: Forced include not found: /build/cirrocumulus-1.1.57/build
  error: subprocess-exited-with-error
```
why do you try me so?

      raise FileNotFoundError(msg)
  FileNotFoundError: Forced include not found: /build/cirrocumulus-1.1.57/build
  error: subprocess-exited-with-error

why do you try me so?

In reply to @truh:matrix.org
For ModuleNotFoundError: No module named 'test.default', { include = "test" }, should be good enough

            Traceback (most recent call last):
              File "/nix/store/mbkyadwnalw99bbvcdfgw4gsjhpzxcgx-python3.12-test-0.0.0/bin/.start-wrapped", line 6, in <module>
                from bin.start import main
              File "/nix/store/mbkyadwnalw99bbvcdfgw4gsjhpzxcgx-python3.12-test-0.0.0/lib/python3.12/site-packages/bin/start.py", line 5, in <module>
                from test.default import get_homedir
            ModuleNotFoundError: No module named 'test.default'

Still the same error with : packages = [{ include = "test" }, { include = "bin" }]

In reply to @tyberius_prime:matrix.org
but that one has me utterly stumped.

what's that for

            ] ++ pkgs.lib.mapAttrsToList
              (name: value:
                pkgs.writers.writePython3Bin "${name}" { flakeIgnore = [ "E401" "E501" ]; } ''
                  import sys, importlib
                  mod, attr = "${value}".split(":", 1)
                  sys.exit(getattr(importlib.import_module(mod), attr)())
                ''
              )
              (builtins.fromTOML (builtins.readFile ./pyproject.toml)).tool.poetry.scripts;

mkPoetryApplication should create bins for the scripts

In reply to @truh:matrix.org

what's that for

            ] ++ pkgs.lib.mapAttrsToList
              (name: value:
                pkgs.writers.writePython3Bin "${name}" { flakeIgnore = [ "E401" "E501" ]; } ''
                  import sys, importlib
                  mod, attr = "${value}".split(":", 1)
                  sys.exit(getattr(importlib.import_module(mod), attr)())
                ''
              )
              (builtins.fromTOML (builtins.readFile ./pyproject.toml)).tool.poetry.scripts;

mkPoetryApplication should create bins for the scripts

Found it in a Github issue. And it was efficient at the time I tested it to remove the bin error.
For what I can remember.

Without this part, I get the following error :

Traceback (most recent call last):
              File "/nix/store/mpn2dgz63imrk5j91ifk3y2nsq5mk9a2-python3.12-test-0.0.0/bin/.start-wrapped", line 6, in <module>
                from bin.start import main
            ModuleNotFoundError: No module named 'bin'

In reply to @truh:matrix.org

what's that for

            ] ++ pkgs.lib.mapAttrsToList
              (name: value:
                pkgs.writers.writePython3Bin "${name}" { flakeIgnore = [ "E401" "E501" ]; } ''
                  import sys, importlib
                  mod, attr = "${value}".split(":", 1)
                  sys.exit(getattr(importlib.import_module(mod), attr)())
                ''
              )
              (builtins.fromTOML (builtins.readFile ./pyproject.toml)).tool.poetry.scripts;

mkPoetryApplication should create bins for the scripts

Found it in a Github issue. And it was efficient at the time I tested it to remove the bin error.
For what I can remember.

I removed it and sill get the error about test.default missing.