Hey! I'm wondering how to correctly set up a devShell using poetry2nix. Previously, I was using:

      devShells.default = pkgs.mkShell {
        packages = with pkgs; [ poetry ];
        LD_LIBRARY_PATH = "${pkgs.stdenv.cc.cc.lib}/lib/";
      };

together with

PYTHON_VERSION=$(python --version | awk '{print $2}' | cut -d. -f1-2 | tr -d '\n')
export PYTHONPATH=$PYTHONPATH:$(poetry env info --path)/lib/python${PYTHON_VERSION}/site-packages

in my .envrc, but I was hoping to get the same effect more gracefully using mkPoetryEnv, so I tried:

      devShells.default = mkPoetryEnv {
        projectDir = ./.;
      };

but for some reason the dependencies aren't available:

» grep pandas pyproject.toml
pandas = "2.1.3"

» python -c "import pandas"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas'

Any idea what I'm doing wrong?

Hey! Whats going on with ipympl?
I've encountered a problem with the ipympl library.

First I added ipympl as a dependency:

poetry add ipympl
Using version ^0.9.4 for ipympl

Then, when I tried to enter the development environment with the command nix develop I received the following message:

nix develop
warning: Git tree '/home/user/test-jupyter' is dirty
error: builder for '/nix/store/yn9990741wv439qncd2wpl2pdbwkjzyz-python3.12-ipympl-0.9.4.drv' failed with exit code 2;
       last 10 log lines:
       >   File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
       >   File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
       >   File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
       >   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
       >   File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
       >   File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
       >   File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
       > ModuleNotFoundError: No module named 'hatchling'

I tried to override ipympl:

myapp = mkPoetryApplication {
    projectDir = self;
    overrides = withDefaults (final: prev: {
        ipympl = prev.ipympl.overridePythonAttrs (old: {
            nativeBuildInputs = old.nativeBuildInputs or [ ] ++ [ final.hatchling ];
        });
    });
};

But it was unsuccessful:

nix develop
warning: Git tree '/home/user/test-jupyter' is dirty
error: builder for '/nix/store/3siq2h2ggxhs4cs2bqr4caczq4vcl15r-python3.12-ipympl-0.9.4.drv' failed with exit code 1;
       last 10 log lines:
       >   full command: /nix/store/z7xxy35k7620hs6fn6la5fg2lgklv72l-python3-3.12.4/bin/python3.12 /nix/store/9gd8kj409msfv599456ywbyi97izlbr4-python3.12-pip-24.0/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /build/tmpwmmwvne3
       >   cwd: /build/ipympl-0.9.4
       >   Preparing metadata (pyproject.toml) ... error
       > error: metadata-generation-failed
       >
       > × Encountered error while generating package metadata.
       > ╰─> See above for output.
       >
       > note: This is an issue with the package mentioned above, not pip.
       > hint: See above for details.
       For full logs, run 'nix log /nix/store/3siq2h2ggxhs4cs2bqr4caczq4vcl15r-python3.12-ipympl-0.9.4.drv'.
error: 1 dependencies of derivation '/nix/store/26zp4sb66agpjwhw7y4pdr5rl9cwwgh1-nix-shell-env.drv' failed to build

I found https://github.com/tweag/jupyenv/blob/0c86802aaa3ffd3e48c6f0e7403031c9168a8be2/lib/overrides.nix#L34C74-L34C95 that there is a hatch-jupyter-builder.

This led to the following error:
ValueError: Aborting. Could not find cmd (jlpm) in path. If command is not expected to be in user's path, use an absolute path.

In reply to @lebastr:matrix.org

Hey! Whats going on with ipympl?
I've encountered a problem with the ipympl library.

First I added ipympl as a dependency:

poetry add ipympl
Using version ^0.9.4 for ipympl

Then, when I tried to enter the development environment with the command nix develop I received the following message:

nix develop
warning: Git tree '/home/user/test-jupyter' is dirty
error: builder for '/nix/store/yn9990741wv439qncd2wpl2pdbwkjzyz-python3.12-ipympl-0.9.4.drv' failed with exit code 2;
       last 10 log lines:
       >   File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
       >   File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
       >   File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
       >   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
       >   File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
       >   File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
       >   File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
       > ModuleNotFoundError: No module named 'hatchling'

I tried to override ipympl:

myapp = mkPoetryApplication {
    projectDir = self;
    overrides = withDefaults (final: prev: {
        ipympl = prev.ipympl.overridePythonAttrs (old: {
            nativeBuildInputs = old.nativeBuildInputs or [ ] ++ [ final.hatchling ];
        });
    });
};

But it was unsuccessful:

nix develop
warning: Git tree '/home/user/test-jupyter' is dirty
error: builder for '/nix/store/3siq2h2ggxhs4cs2bqr4caczq4vcl15r-python3.12-ipympl-0.9.4.drv' failed with exit code 1;
       last 10 log lines:
       >   full command: /nix/store/z7xxy35k7620hs6fn6la5fg2lgklv72l-python3-3.12.4/bin/python3.12 /nix/store/9gd8kj409msfv599456ywbyi97izlbr4-python3.12-pip-24.0/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /build/tmpwmmwvne3
       >   cwd: /build/ipympl-0.9.4
       >   Preparing metadata (pyproject.toml) ... error
       > error: metadata-generation-failed
       >
       > × Encountered error while generating package metadata.
       > ╰─> See above for output.
       >
       > note: This is an issue with the package mentioned above, not pip.
       > hint: See above for details.
       For full logs, run 'nix log /nix/store/3siq2h2ggxhs4cs2bqr4caczq4vcl15r-python3.12-ipympl-0.9.4.drv'.
error: 1 dependencies of derivation '/nix/store/26zp4sb66agpjwhw7y4pdr5rl9cwwgh1-nix-shell-env.drv' failed to build

It's really weird!

I ended up with this config:

myapp = mkPoetryApplication {
            projectDir = self;
            overrides = withDefaults (final: prev: {
              ipympl = prev.ipympl.overridePythonAttrs (old: {
                  nativeBuildInputs = old.nativeBuildInputs or [ ] ++ [
                  final.hatchling
                  final.hatch-jupyter-builder
                  final.jupyterlab
                  pkgs.nodejs
                ];
              });

            });
          };

And I've got a crazy error message:

Internal Error: EACCES: permission denied, mkdir '/homeless-shelter'
  Error: EACCES: permission denied, mkdir '/homeless-shelter'

WARNING: The directory '/homeless-shelter/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. >
Processing /build/ipympl-0.9.4
  Running command Preparing metadata (pyproject.toml)
  INFO:hatch_jupyter_builder.utils:Running jupyter-builder
  INFO:hatch_jupyter_builder.utils:Building with hatch_jupyter_builder.npm_builder
  INFO:hatch_jupyter_builder.utils:With kwargs: {'path': '.', 'build_cmd': 'build:prod', 'npm': ['jlpm']}
  INFO:hatch_jupyter_builder.utils:Installing build dependencies with npm.  This may take a while...
  INFO:hatch_jupyter_builder.utils:> /nix/store/3ah78dwwli5k9zxwfw7nhdpycm82lp8g-python3.12-jupyterlab-4.2.4/bin/jlpm install
  Internal Error: EACCES: permission denied, mkdir '/homeless-shelter'
  Error: EACCES: permission denied, mkdir '/homeless-shelter'
  Traceback (most recent call last):
    File "/nix/store/9gd8kj409msfv599456ywbyi97izlbr4-python3.12-pip-24.0/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/nix/store/9gd8kj409msfv599456ywbyi97izlbr4-python3.12-pip-24.0/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/nix/store/9gd8kj409msfv599456ywbyi97izlbr4-python3.12-pip-24.0/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
      whl_basename = backend.build_wheel(metadata_directory, config_settings)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/nix/store/nrwbrccny1ycgs0lhv7x4agic3ncxsyn-python3.12-hatchling-1.25.0/lib/python3.12/site-packages/hatchling/build.py", line 58, in build_wheel
      return os.path.basename(next(builder.build(directory=wheel_directory, versions=['standard'])))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/nix/store/nrwbrccny1ycgs0lhv7x4agic3ncxsyn-python3.12-hatchling-1.25.0/lib/python3.12/site-packages/hatchling/builders/plugin/interface.py", line 147, in build
      build_hook.initialize(version, build_data)
    File "/nix/store/qhp7ahvskg59nj49sr99gz78qfqsic34-python3.12-hatch-jupyter-builder-0.9.1/lib/python3.12/site-packages/hatch_jupyter_builder/plugin.py", line 94, in initialize
      raise e
    File "/nix/store/qhp7ahvskg59nj49sr99gz78qfqsic34-python3.12-hatch-jupyter-builder-0.9.1/lib/python3.12/site-packages/hatch_jupyter_builder/plugin.py", line 89, in initialize
      build_func(self.target_name, version, **build_kwargs)
    File "/nix/store/qhp7ahvskg59nj49sr99gz78qfqsic34-python3.12-hatch-jupyter-builder-0.9.1/lib/python3.12/site-packages/hatch_jupyter_builder/utils.py", line 115, in npm_builder
      run([*npm_cmd, "install"], cwd=str(abs_path))
    File "/nix/store/qhp7ahvskg59nj49sr99gz78qfqsic34-python3.12-hatch-jupyter-builder-0.9.1/lib/python3.12/site-packages/hatch_jupyter_builder/utils.py", line 231, in run
      return subprocess.check_call(cmd, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/nix/store/z7xxy35k7620hs6fn6la5fg2lgklv72l-python3-3.12.4/lib/python3.12/subprocess.py", line 413, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/nix/store/3ah78dwwli5k9zxwfw7nhdpycm82lp8g-python3.12-jupyterlab-4.2.4/bin/jlpm', 'install']' returned non-zero exit status 1.
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

In reply to @k900:0upti.me
Try adding preBuild = "export HOME=$(mktemp -d)"

It seems that ipympl was installed successfully.
Thank you!

But it was really complicated

Hey! I tried to override cargoDeps of rpds-py package to support 0.19.1 version.
I have the following code:

myapp = mkPoetryApplication { 
            projectDir = self; 
            overrides = p2nix.overrides.withDefaults (final: prev: {
              rpds-py = prev.rpds-py.overridePythonAttrs (old: 
                lib.optionalAttrs (old.version == "0.19.1") ( lib.warn "version == 0.19.1" {
                  cargoDeps = lib.warn "override cargoDeps" (pkgs.rustPlatform.fetchCargoTarball {
                    inherit (old) src;
                    name = "${old.pname}-${old.version}";
                    hash = lib.warn "new hash was used in cargoDeps" "sha256-qIXdoCEVGCGUnTicZp4bUTJyGpFy9dwWY03lXUbxiHg=";
                  });

                  cargoFakeDeps = lib.warn "override cargoFakeDeps" (pkgs.rustPlatform.fetchCargoTarball {
                    inherit (old) src;
                    name = "${old.pname}-${old.version}";
                    hash = lib.warn "new hash was used in cargoFakeDeps" "sha256-qIXdoCEVGCGUnTicZp4bUTJyGpFy9dwWY03lXUbxiHg=";
                  });
                })
              );
            });
          };

I see that cargoFakeDeps was evaluated but cargoDeps wasn't.
This is the log output I received:

[user@daily:~/tmp/jupyter-test]$ nix develop
warning: Git tree '/home/user/tmp/jupyter-test' is dirty
trace: warning: version == 0.19.1
trace: warning: Unknown rpds-py version: '0.19.1'. Please update getCargoHash.
trace: warning: override cargoFakeDeps
trace: warning: new hash was used in cargoFakeDeps
error: hash mismatch in fixed-output derivation '/nix/store/3m2xsv79lvb6pv4f3sd5nyfv8sl0byrb-rpds-py-0.19.1-vendor.tar.gz.drv':
         specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-qIXdoCEVGCGUnTicZp4bUTJyGpFy9dwWY03lXUbxiHg=
error: 1 dependencies of derivation '/nix/store/8frwibaaxwga13d1vsv4b3m17ynhlrrr-python3.11-rpds-py-0.19.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/bjlcbjsy5cs57pm2qmaw2b7whgd0lrxj-python3.11-jsonschema-4.23.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/45j5816fkpznmby4jjaz8w8f1vb28ar9-python3.11-referencing-0.35.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/qwf87z1fjbf1rm4ml2rffrp7q9gkbbz6-python3.11-jupyter-events-0.10.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/y8m4si7ikw0s5zdzwml1cwvf6pvif181-python3.11-jupyterlab-server-2.27.3.drv' failed to build
error: 1 dependencies of derivation '/nix/store/a7sh6l94yql8g89x81ndsc7rnpw1l3qh-python3.11-nbformat-5.10.4.drv' failed to build
error: 1 dependencies of derivation '/nix/store/rsr6ll0kxdlgm09356y8hamlpzyzqx2b-python3.11-nbconvert-7.16.4.drv' failed to build
error: 1 dependencies of derivation '/nix/store/qzljggjx929jv9pj73jdy5x5wybsm0ic-python3.11-jupyter-1.0.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/76h3qhf1ijj1zyvmz0753a8qrznm06v9-nix-shell-env.drv' failed to build

What did I do wrong?

In reply to @lebastr:matrix.org

Hey! I tried to override cargoDeps of rpds-py package to support 0.19.1 version.
I have the following code:

myapp = mkPoetryApplication { 
            projectDir = self; 
            overrides = p2nix.overrides.withDefaults (final: prev: {
              rpds-py = prev.rpds-py.overridePythonAttrs (old: 
                lib.optionalAttrs (old.version == "0.19.1") ( lib.warn "version == 0.19.1" {
                  cargoDeps = lib.warn "override cargoDeps" (pkgs.rustPlatform.fetchCargoTarball {
                    inherit (old) src;
                    name = "${old.pname}-${old.version}";
                    hash = lib.warn "new hash was used in cargoDeps" "sha256-qIXdoCEVGCGUnTicZp4bUTJyGpFy9dwWY03lXUbxiHg=";
                  });

                  cargoFakeDeps = lib.warn "override cargoFakeDeps" (pkgs.rustPlatform.fetchCargoTarball {
                    inherit (old) src;
                    name = "${old.pname}-${old.version}";
                    hash = lib.warn "new hash was used in cargoFakeDeps" "sha256-qIXdoCEVGCGUnTicZp4bUTJyGpFy9dwWY03lXUbxiHg=";
                  });
                })
              );
            });
          };

I see that cargoFakeDeps was evaluated but cargoDeps wasn't.
This is the log output I received:

[user@daily:~/tmp/jupyter-test]$ nix develop
warning: Git tree '/home/user/tmp/jupyter-test' is dirty
trace: warning: version == 0.19.1
trace: warning: Unknown rpds-py version: '0.19.1'. Please update getCargoHash.
trace: warning: override cargoFakeDeps
trace: warning: new hash was used in cargoFakeDeps
error: hash mismatch in fixed-output derivation '/nix/store/3m2xsv79lvb6pv4f3sd5nyfv8sl0byrb-rpds-py-0.19.1-vendor.tar.gz.drv':
         specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-qIXdoCEVGCGUnTicZp4bUTJyGpFy9dwWY03lXUbxiHg=
error: 1 dependencies of derivation '/nix/store/8frwibaaxwga13d1vsv4b3m17ynhlrrr-python3.11-rpds-py-0.19.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/bjlcbjsy5cs57pm2qmaw2b7whgd0lrxj-python3.11-jsonschema-4.23.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/45j5816fkpznmby4jjaz8w8f1vb28ar9-python3.11-referencing-0.35.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/qwf87z1fjbf1rm4ml2rffrp7q9gkbbz6-python3.11-jupyter-events-0.10.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/y8m4si7ikw0s5zdzwml1cwvf6pvif181-python3.11-jupyterlab-server-2.27.3.drv' failed to build
error: 1 dependencies of derivation '/nix/store/a7sh6l94yql8g89x81ndsc7rnpw1l3qh-python3.11-nbformat-5.10.4.drv' failed to build
error: 1 dependencies of derivation '/nix/store/rsr6ll0kxdlgm09356y8hamlpzyzqx2b-python3.11-nbconvert-7.16.4.drv' failed to build
error: 1 dependencies of derivation '/nix/store/qzljggjx929jv9pj73jdy5x5wybsm0ic-python3.11-jupyter-1.0.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/76h3qhf1ijj1zyvmz0753a8qrznm06v9-nix-shell-env.drv' failed to build

What did I do wrong?

I found problem.
My overlay was overwritten by default:

withDefaults = overlay: [
      overlay
      self.defaultPoetryOverrides
    ];

Hi all. Is there a best practice I'm missing for providing poetry itself in a dev shell? I've been using (slightly customised) versions of pkgs.poetry as a buildInputs / packages entry, but was wondering if poetry2nix exposes its version cleverly maybe.

(Motivation: I find this is needed to do poetry lock --no-update or poetry add foo --lock etc etc, and obviously don't want to use a global poetry version)

Hello,

I am trying to convert a project with poetry2nix.
The project has a web server launch via a gunicorn with Popen.

I have errors related to the fact that gunircorn could not import library such as filtetype.

[2024-08-01 01:04:01 +0200] [1248518] [INFO] Starting gunicorn 22.0.0
[2024-08-01 01:04:01 +0200] [1248518] [INFO] Listening at: http://0.0.0.0:5100 (1248518)
[2024-08-01 01:04:01 +0200] [1248518] [INFO] Using worker: sync
[2024-08-01 01:04:01 +0200] [1248521] [INFO] Booting worker with pid: 1248521
[2024-08-01 01:04:01 +0200] [1248521] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
    worker.init_process()
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
                ^^^^^^^^^^^^^^^
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
                    ^^^^^^^^^^^
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
           ^^^^^^^^^^^^^^^^^^^ File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/util.py", line 371, in import_app
    mod = importlib.import_module(module)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/jms4z7lzzwnv6gv3y0795365haicmh8m-python3-3.12.4/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/user/Repository/lookyloo/website/web/__init__.py", line 17, in <module>
    import filetype  # type: ignore[import-untyped]
    ^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'filetype'
[2024-08-01 01:04:01 +0200] [1248521] [INFO] Worker exiting (pid: 1248521)
[2024-08-01 01:04:01 +0200] [1248518] [ERROR] Worker (pid:1248521) exited with code 3
[2024-08-01 01:04:01 +0200] [1248518] [ERROR] Shutting down: Master
[2024-08-01 01:04:01 +0200] [1248518] [ERROR] Reason: Worker failed to boot.

The project is the following one https://github.com/Lookyloo/lookyloo.
With the following patch applied :

diff --git a/.gitignore b/.gitignore
index 3b2b988..7ca0859 100644
--- a/.gitignore
+++ b/.gitignore
@@ -151,3 +151,6 @@ sanejs
 urlhaus
 urlscan
 vt_url
+
+# Nix
+result
diff --git a/cache/run_redis.sh b/cache/run_redis.sh
index 8daf6d8..8a70742 100755
--- a/cache/run_redis.sh
+++ b/cache/run_redis.sh
@@ -1,13 +1,13 @@
-#!/bin/bash
+#!/usr/bin/env bash

 set -e
 set -x

-if [ -f  ../../valkey/src/valkey-server ]; then
-    ../../valkey/src/valkey-server ./cache.conf
+if [ -f ../../valkey/src/valkey-server ]; then
+  ../../valkey/src/valkey-server ./cache.conf
 elif [ -f ../../redis/src/redis-server ]; then
-    ../../redis/src/redis-server ./cache.conf
+  ../../redis/src/redis-server ./cache.conf
 else
-    echo "Warning: using system redis-server. Valkey-server or redis-server from source is recommended." >&2
-    /usr/bin/redis-server ./cache.conf
+  echo "Warning: using system redis-server. Valkey-server or redis-server from source is recommended." >&2
+  redis-server ./cache.conf
 fi
diff --git a/indexing/run_redis.sh b/indexing/run_redis.sh
index d6924a0..7107500 100755
--- a/indexing/run_redis.sh
+++ b/indexing/run_redis.sh
@@ -1,13 +1,13 @@
-#!/bin/bash
+#!/usr/bin/env bash

 set -e
 set -x

-if [ -f  ../../valkey/src/valkey-server ]; then
-    ../../valkey/src/valkey-server ./indexing.conf
+if [ -f ../../valkey/src/valkey-server ]; then
+  ../../valkey/src/valkey-server ./indexing.conf
 elif [ -f ../../redis/src/redis-server ]; then
-    ../../redis/src/redis-server ./indexing.conf
+  ../../redis/src/redis-server ./indexing.conf
 else
-    echo "Warning: using system redis-server. Valkey-server or redis-server from source is recommended." >&2
-    /usr/bin/redis-server ./indexing.conf
+  echo "Warning: using system redis-server. Valkey-server or redis-server from source is recommended." >&2
+  redis-server ./indexing.conf
 fi
diff --git a/pyproject.toml b/pyproject.toml
index 7614ce8..2ded159 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -21,6 +21,11 @@ classifiers = [
     'Topic :: Internet',
 ]

+packages = [
+{include = "lookyloo"},
+{include = "bin"},
+]
+
 [tool.poetry.scripts]
 start = "bin.start:main"
 stop = "bin.stop:main"

The flake is :

{
  description = "Application packaged using poetry2nix";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    poetry2nix = {
      #url = "github:nix-community/poetry2nix";
      url = "git+https://gitea.com/oven_spinout988/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    flake-utils.lib.eachDefaultSystem
    (system:
      let
        pkgs = nixpkgs.legacyPackages.${system}.extend poetry2nix.overlays.default;

        projectDir = ./.;

        # NOTE: add non python packages, needed at runtime, here
        runtimePackages = with pkgs; [
          redis
          ffmpeg
          python3
        ];

        # NOTE: add non python packages, only needed for development, here
        devPackages = with pkgs; [
          pkgs.python3
          pkgs.poetry
        ];

        patches = [
          ./nix/cache.patch
          ./nix/indexing.patch
        ];

        pypkgs-build-requirements = {
          fsspec = [ "hatchling" "hatch-vcs" ];
          har2tree = [ "poetry-core" ];
          pydeep2 = [ "setuptools-scm" "ssdeep" pkgs.ssdeep pkgs.libtool pkgs.automake ];
          vt-py = [ "setuptools" "pytest-runner" ];
          pyeupi = [ "poetry-core" ];
          pyfaup = [ "setuptools" ];
          types-cffi = [ "setuptools" ];
          pypdns = [ "poetry-core" ];
          passivetotal = [ "setuptools" ];
          pyhashlookup = [ "poetry-core" ];
          pyipasnhistory = [ "poetry-core" ];
          pylookyloo = [ "poetry-core" ];
          pymisp = [ "poetry-core" ];
          pylookyloomonitoring = [ "poetry-core" ];
          pypandora = [ "poetry-core" ];
          pyphishtanklookup = [ "poetry-core" ];
          pysanejs = [ "poetry-core" ];
          pysecuritytxt = [ "poetry-core" ];
          pytaxonomies = [ "poetry-core" ];
          flask = [ "setuptools" ];
          ssdeep = [ "setuptools" ];
          lxml = [ pkgs.libz ];
          filetype = [ "setuptools" ];
        };
        p2n-overrides = pkgs.poetry2nix.overrides.withDefaults
          #p2n-overrides = defaultPoetryOverrides.extend
          (final: prev:
            builtins.mapAttrs
            (package: build-requirements:
              (builtins.getAttr package prev).overridePythonAttrs (old: {
                buildInputs = (old.buildInputs or [ ]) ++ (builtins.map (pkg: if builtins.isString pkg then builtins.getAttr pkg prev else pkg) build-requirements);
              })
            )

            pypkgs-build-requirements
          );

        propagatedBuildInputs = [
          pkgs.ffmpeg
          pkgs.redis
          # pkgs.python3
        ];

        lookylooApp = pkgs.poetry2nix.mkPoetryApplication
          {
            inherit projectDir;
            inherit propagatedBuildInputs;
            overrides = p2n-overrides;
          };
      in
        {
        packages = {
          default = lookylooApp;
        };

        apps = {
          start = {
            type = "app";
            program = "${lookylooApp}/bin/start";
          };

          start_website = {
            type = "app";
            program = "${lookylooApp}/bin/start_website";
          };

          stop = {
            type = "app";
            program = "${lookylooApp}/bin/stop";
          };
        };

        devShells.default =
          pkgs.mkShell
          {
            # inputsFrom = [ self.packages.${system}.lookyloo ];

            packages = [
              lookylooApp.dependencyEnv
            ]
            ++ runtimePackages
            ++ devPackages;

            shellHook = ''
                  export LOOKYLOO_HOME="$(pwd)";
                  export PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=true

                  echo "[+] Welcome to this nix shell"
                  echo "- Python $(python --version)"
                  '';
          };
      });
}

It is working with the dev environment, what am I doing wrong ?

Hello,

I am trying to convert a project with poetry2nix.
The project has a web server launch via a gunicorn with Popen.
The webserver configuration is under website/web where scripts are in bin and the main library is under lookyloo.

I have errors related to the fact that gunircorn could not import library such as filtetype.

[2024-08-01 01:04:01 +0200] [1248518] [INFO] Starting gunicorn 22.0.0
[2024-08-01 01:04:01 +0200] [1248518] [INFO] Listening at: http://0.0.0.0:5100 (1248518)
[2024-08-01 01:04:01 +0200] [1248518] [INFO] Using worker: sync
[2024-08-01 01:04:01 +0200] [1248521] [INFO] Booting worker with pid: 1248521
[2024-08-01 01:04:01 +0200] [1248521] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
    worker.init_process()
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
                ^^^^^^^^^^^^^^^
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
                    ^^^^^^^^^^^
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
           ^^^^^^^^^^^^^^^^^^^ File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/util.py", line 371, in import_app
    mod = importlib.import_module(module)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/jms4z7lzzwnv6gv3y0795365haicmh8m-python3-3.12.4/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/user/Repository/lookyloo/website/web/__init__.py", line 17, in <module>
    import filetype  # type: ignore[import-untyped]
    ^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'filetype'
[2024-08-01 01:04:01 +0200] [1248521] [INFO] Worker exiting (pid: 1248521)
[2024-08-01 01:04:01 +0200] [1248518] [ERROR] Worker (pid:1248521) exited with code 3
[2024-08-01 01:04:01 +0200] [1248518] [ERROR] Shutting down: Master
[2024-08-01 01:04:01 +0200] [1248518] [ERROR] Reason: Worker failed to boot.

The project is the following one https://github.com/Lookyloo/lookyloo.
With the following patch applied :

diff --git a/.gitignore b/.gitignore
index 3b2b988..7ca0859 100644
--- a/.gitignore
+++ b/.gitignore
@@ -151,3 +151,6 @@ sanejs
 urlhaus
 urlscan
 vt_url
+
+# Nix
+result
diff --git a/cache/run_redis.sh b/cache/run_redis.sh
index 8daf6d8..8a70742 100755
--- a/cache/run_redis.sh
+++ b/cache/run_redis.sh
@@ -1,13 +1,13 @@
-#!/bin/bash
+#!/usr/bin/env bash

 set -e
 set -x

-if [ -f  ../../valkey/src/valkey-server ]; then
-    ../../valkey/src/valkey-server ./cache.conf
+if [ -f ../../valkey/src/valkey-server ]; then
+  ../../valkey/src/valkey-server ./cache.conf
 elif [ -f ../../redis/src/redis-server ]; then
-    ../../redis/src/redis-server ./cache.conf
+  ../../redis/src/redis-server ./cache.conf
 else
-    echo "Warning: using system redis-server. Valkey-server or redis-server from source is recommended." >&2
-    /usr/bin/redis-server ./cache.conf
+  echo "Warning: using system redis-server. Valkey-server or redis-server from source is recommended." >&2
+  redis-server ./cache.conf
 fi
diff --git a/indexing/run_redis.sh b/indexing/run_redis.sh
index d6924a0..7107500 100755
--- a/indexing/run_redis.sh
+++ b/indexing/run_redis.sh
@@ -1,13 +1,13 @@
-#!/bin/bash
+#!/usr/bin/env bash

 set -e
 set -x

-if [ -f  ../../valkey/src/valkey-server ]; then
-    ../../valkey/src/valkey-server ./indexing.conf
+if [ -f ../../valkey/src/valkey-server ]; then
+  ../../valkey/src/valkey-server ./indexing.conf
 elif [ -f ../../redis/src/redis-server ]; then
-    ../../redis/src/redis-server ./indexing.conf
+  ../../redis/src/redis-server ./indexing.conf
 else
-    echo "Warning: using system redis-server. Valkey-server or redis-server from source is recommended." >&2
-    /usr/bin/redis-server ./indexing.conf
+  echo "Warning: using system redis-server. Valkey-server or redis-server from source is recommended." >&2
+  redis-server ./indexing.conf
 fi
diff --git a/pyproject.toml b/pyproject.toml
index 7614ce8..2ded159 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -21,6 +21,11 @@ classifiers = [
     'Topic :: Internet',
 ]

+packages = [
+{include = "lookyloo"},
+{include = "bin"},
+]
+
 [tool.poetry.scripts]
 start = "bin.start:main"
 stop = "bin.stop:main"

The flake is :

{
  description = "Application packaged using poetry2nix";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    poetry2nix = {
      #url = "github:nix-community/poetry2nix";
      url = "git+https://gitea.com/oven_spinout988/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    flake-utils.lib.eachDefaultSystem
    (system:
      let
        pkgs = nixpkgs.legacyPackages.${system}.extend poetry2nix.overlays.default;

        projectDir = ./.;

        # NOTE: add non python packages, needed at runtime, here
        runtimePackages = with pkgs; [
          redis
          ffmpeg
          python3
        ];

        # NOTE: add non python packages, only needed for development, here
        devPackages = with pkgs; [
          pkgs.python3
          pkgs.poetry
        ];

        patches = [
          ./nix/cache.patch
          ./nix/indexing.patch
        ];

        pypkgs-build-requirements = {
          fsspec = [ "hatchling" "hatch-vcs" ];
          har2tree = [ "poetry-core" ];
          pydeep2 = [ "setuptools-scm" "ssdeep" pkgs.ssdeep pkgs.libtool pkgs.automake ];
          vt-py = [ "setuptools" "pytest-runner" ];
          pyeupi = [ "poetry-core" ];
          pyfaup = [ "setuptools" ];
          types-cffi = [ "setuptools" ];
          pypdns = [ "poetry-core" ];
          passivetotal = [ "setuptools" ];
          pyhashlookup = [ "poetry-core" ];
          pyipasnhistory = [ "poetry-core" ];
          pylookyloo = [ "poetry-core" ];
          pymisp = [ "poetry-core" ];
          pylookyloomonitoring = [ "poetry-core" ];
          pypandora = [ "poetry-core" ];
          pyphishtanklookup = [ "poetry-core" ];
          pysanejs = [ "poetry-core" ];
          pysecuritytxt = [ "poetry-core" ];
          pytaxonomies = [ "poetry-core" ];
          flask = [ "setuptools" ];
          ssdeep = [ "setuptools" ];
          lxml = [ pkgs.libz ];
          filetype = [ "setuptools" ];
        };
        p2n-overrides = pkgs.poetry2nix.overrides.withDefaults
          #p2n-overrides = defaultPoetryOverrides.extend
          (final: prev:
            builtins.mapAttrs
            (package: build-requirements:
              (builtins.getAttr package prev).overridePythonAttrs (old: {
                buildInputs = (old.buildInputs or [ ]) ++ (builtins.map (pkg: if builtins.isString pkg then builtins.getAttr pkg prev else pkg) build-requirements);
              })
            )

            pypkgs-build-requirements
          );

        propagatedBuildInputs = [
          pkgs.ffmpeg
          pkgs.redis
          # pkgs.python3
        ];

        lookylooApp = pkgs.poetry2nix.mkPoetryApplication
          {
            inherit projectDir;
            inherit propagatedBuildInputs;
            overrides = p2n-overrides;
          };
      in
        {
        packages = {
          default = lookylooApp;
        };

        apps = {
          start = {
            type = "app";
            program = "${lookylooApp}/bin/start";
          };

          start_website = {
            type = "app";
            program = "${lookylooApp}/bin/start_website";
          };

          stop = {
            type = "app";
            program = "${lookylooApp}/bin/stop";
          };
        };

        devShells.default =
          pkgs.mkShell
          {
            # inputsFrom = [ self.packages.${system}.lookyloo ];

            packages = [
              lookylooApp.dependencyEnv
            ]
            ++ runtimePackages
            ++ devPackages;

            shellHook = ''
                  export LOOKYLOO_HOME="$(pwd)";
                  export PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=true

                  echo "[+] Welcome to this nix shell"
                  echo "- Python $(python --version)"
                  '';
          };
      });
}

It is working with the dev environment, what am I doing wrong ?

Hello,

I am trying to convert a project with poetry2nix.
The project has a web server launch via a gunicorn with Popen.
The webserver configuration is under website/web where scripts are in bin and the main library is under lookyloo.

I have errors related to the fact that gunircorn could not import library such as filtetype.

[2024-08-01 01:04:01 +0200] [1248518] [INFO] Starting gunicorn 22.0.0
[2024-08-01 01:04:01 +0200] [1248518] [INFO] Listening at: http://0.0.0.0:5100 (1248518)
[2024-08-01 01:04:01 +0200] [1248518] [INFO] Using worker: sync
[2024-08-01 01:04:01 +0200] [1248521] [INFO] Booting worker with pid: 1248521
[2024-08-01 01:04:01 +0200] [1248521] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
    worker.init_process()
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
                ^^^^^^^^^^^^^^^
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
                    ^^^^^^^^^^^
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
           ^^^^^^^^^^^^^^^^^^^ File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/util.py", line 371, in import_app
    mod = importlib.import_module(module)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/jms4z7lzzwnv6gv3y0795365haicmh8m-python3-3.12.4/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/user/Repository/lookyloo/website/web/__init__.py", line 17, in <module>
    import filetype  # type: ignore[import-untyped]
    ^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'filetype'
[2024-08-01 01:04:01 +0200] [1248521] [INFO] Worker exiting (pid: 1248521)
[2024-08-01 01:04:01 +0200] [1248518] [ERROR] Worker (pid:1248521) exited with code 3
[2024-08-01 01:04:01 +0200] [1248518] [ERROR] Shutting down: Master
[2024-08-01 01:04:01 +0200] [1248518] [ERROR] Reason: Worker failed to boot.

The project is the following one https://github.com/Lookyloo/lookyloo.
With the following patch applied :

diff --git a/.gitignore b/.gitignore
index 3b2b988..7ca0859 100644
--- a/.gitignore
+++ b/.gitignore
@@ -151,3 +151,6 @@ sanejs
 urlhaus
 urlscan
 vt_url
+
+# Nix
+result
diff --git a/cache/run_redis.sh b/cache/run_redis.sh
index 8daf6d8..8a70742 100755
--- a/cache/run_redis.sh
+++ b/cache/run_redis.sh
@@ -1,13 +1,13 @@
-#!/bin/bash
+#!/usr/bin/env bash

 set -e
 set -x

-if [ -f  ../../valkey/src/valkey-server ]; then
-    ../../valkey/src/valkey-server ./cache.conf
+if [ -f ../../valkey/src/valkey-server ]; then
+  ../../valkey/src/valkey-server ./cache.conf
 elif [ -f ../../redis/src/redis-server ]; then
-    ../../redis/src/redis-server ./cache.conf
+  ../../redis/src/redis-server ./cache.conf
 else
-    echo "Warning: using system redis-server. Valkey-server or redis-server from source is recommended." >&2
-    /usr/bin/redis-server ./cache.conf
+  echo "Warning: using system redis-server. Valkey-server or redis-server from source is recommended." >&2
+  redis-server ./cache.conf
 fi
diff --git a/indexing/run_redis.sh b/indexing/run_redis.sh
index d6924a0..7107500 100755
--- a/indexing/run_redis.sh
+++ b/indexing/run_redis.sh
@@ -1,13 +1,13 @@
-#!/bin/bash
+#!/usr/bin/env bash

 set -e
 set -x

-if [ -f  ../../valkey/src/valkey-server ]; then
-    ../../valkey/src/valkey-server ./indexing.conf
+if [ -f ../../valkey/src/valkey-server ]; then
+  ../../valkey/src/valkey-server ./indexing.conf
 elif [ -f ../../redis/src/redis-server ]; then
-    ../../redis/src/redis-server ./indexing.conf
+  ../../redis/src/redis-server ./indexing.conf
 else
-    echo "Warning: using system redis-server. Valkey-server or redis-server from source is recommended." >&2
-    /usr/bin/redis-server ./indexing.conf
+  echo "Warning: using system redis-server. Valkey-server or redis-server from source is recommended." >&2
+  redis-server ./indexing.conf
 fi
diff --git a/pyproject.toml b/pyproject.toml
index 7614ce8..2ded159 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -21,6 +21,11 @@ classifiers = [
     'Topic :: Internet',
 ]

+packages = [
+{include = "lookyloo"},
+{include = "bin"},
+]
+
 [tool.poetry.scripts]
 start = "bin.start:main"
 stop = "bin.stop:main"

The flake is :

{
  description = "Application packaged using poetry2nix";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    poetry2nix = {
      #url = "github:nix-community/poetry2nix";
      url = "git+https://gitea.com/oven_spinout988/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    flake-utils.lib.eachDefaultSystem
    (system:
      let
        pkgs = nixpkgs.legacyPackages.${system}.extend poetry2nix.overlays.default;

        projectDir = ./.;

        # NOTE: add non python packages, needed at runtime, here
        runtimePackages = with pkgs; [
          redis
          ffmpeg
          python3
        ];

        # NOTE: add non python packages, only needed for development, here
        devPackages = with pkgs; [
          pkgs.python3
          pkgs.poetry
        ];

        patches = [
          ./nix/cache.patch
          ./nix/indexing.patch
        ];

        pypkgs-build-requirements = {
          fsspec = [ "hatchling" "hatch-vcs" ];
          har2tree = [ "poetry-core" ];
          pydeep2 = [ "setuptools-scm" "ssdeep" pkgs.ssdeep pkgs.libtool pkgs.automake ];
          vt-py = [ "setuptools" "pytest-runner" ];
          pyeupi = [ "poetry-core" ];
          pyfaup = [ "setuptools" ];
          types-cffi = [ "setuptools" ];
          pypdns = [ "poetry-core" ];
          passivetotal = [ "setuptools" ];
          pyhashlookup = [ "poetry-core" ];
          pyipasnhistory = [ "poetry-core" ];
          pylookyloo = [ "poetry-core" ];
          pymisp = [ "poetry-core" ];
          pylookyloomonitoring = [ "poetry-core" ];
          pypandora = [ "poetry-core" ];
          pyphishtanklookup = [ "poetry-core" ];
          pysanejs = [ "poetry-core" ];
          pysecuritytxt = [ "poetry-core" ];
          pytaxonomies = [ "poetry-core" ];
          flask = [ "setuptools" ];
          ssdeep = [ "setuptools" ];
          lxml = [ pkgs.libz ];
          filetype = [ "setuptools" ];
        };
        p2n-overrides = pkgs.poetry2nix.overrides.withDefaults
          #p2n-overrides = defaultPoetryOverrides.extend
          (final: prev:
            builtins.mapAttrs
            (package: build-requirements:
              (builtins.getAttr package prev).overridePythonAttrs (old: {
                buildInputs = (old.buildInputs or [ ]) ++ (builtins.map (pkg: if builtins.isString pkg then builtins.getAttr pkg prev else pkg) build-requirements);
              })
            )

            pypkgs-build-requirements
          );

        propagatedBuildInputs = [
          pkgs.ffmpeg
          pkgs.redis
          # pkgs.python3
        ];

        lookylooApp = pkgs.poetry2nix.mkPoetryApplication
          {
            inherit projectDir;
            inherit propagatedBuildInputs;
            overrides = p2n-overrides;
          };
      in
        {
        packages = {
          default = lookylooApp;
        };

        apps = {
          start = {
            type = "app";
            program = "${lookylooApp}/bin/start";
          };

          start_website = {
            type = "app";
            program = "${lookylooApp}/bin/start_website";
          };

          stop = {
            type = "app";
            program = "${lookylooApp}/bin/stop";
          };
        };

        devShells.default =
          pkgs.mkShell
          {
            # inputsFrom = [ self.packages.${system}.lookyloo ];

            packages = [
              lookylooApp.dependencyEnv
            ]
            ++ runtimePackages
            ++ devPackages;

            shellHook = ''
                  export LOOKYLOO_HOME="$(pwd)";
                  export PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=true

                  echo "[+] Welcome to this nix shell"
                  echo "- Python $(python --version)"
                  '';
          };
      });
}

It is working with the dev environment.
Can you help me to find out what I am doing wrong ?

In reply to @oven_spinout988:matrix.org

Hello,

I am trying to convert a project with poetry2nix.
The project has a web server launch via a gunicorn with Popen.
The webserver configuration is under website/web where scripts are in bin and the main library is under lookyloo.

I have errors related to the fact that gunircorn could not import library such as filtetype.

[2024-08-01 01:04:01 +0200] [1248518] [INFO] Starting gunicorn 22.0.0
[2024-08-01 01:04:01 +0200] [1248518] [INFO] Listening at: http://0.0.0.0:5100 (1248518)
[2024-08-01 01:04:01 +0200] [1248518] [INFO] Using worker: sync
[2024-08-01 01:04:01 +0200] [1248521] [INFO] Booting worker with pid: 1248521
[2024-08-01 01:04:01 +0200] [1248521] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
    worker.init_process()
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
                ^^^^^^^^^^^^^^^
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
                    ^^^^^^^^^^^
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
           ^^^^^^^^^^^^^^^^^^^ File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/dwp52sk1jzz0lqy499g5hblp00ns97d0-python3.12-gunicorn-22.0.0/lib/python3.12/site-packages/gunicorn/util.py", line 371, in import_app
    mod = importlib.import_module(module)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/jms4z7lzzwnv6gv3y0795365haicmh8m-python3-3.12.4/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/user/Repository/lookyloo/website/web/__init__.py", line 17, in <module>
    import filetype  # type: ignore[import-untyped]
    ^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'filetype'
[2024-08-01 01:04:01 +0200] [1248521] [INFO] Worker exiting (pid: 1248521)
[2024-08-01 01:04:01 +0200] [1248518] [ERROR] Worker (pid:1248521) exited with code 3
[2024-08-01 01:04:01 +0200] [1248518] [ERROR] Shutting down: Master
[2024-08-01 01:04:01 +0200] [1248518] [ERROR] Reason: Worker failed to boot.

The project is the following one https://github.com/Lookyloo/lookyloo.
With the following patch applied :

diff --git a/.gitignore b/.gitignore
index 3b2b988..7ca0859 100644
--- a/.gitignore
+++ b/.gitignore
@@ -151,3 +151,6 @@ sanejs
 urlhaus
 urlscan
 vt_url
+
+# Nix
+result
diff --git a/cache/run_redis.sh b/cache/run_redis.sh
index 8daf6d8..8a70742 100755
--- a/cache/run_redis.sh
+++ b/cache/run_redis.sh
@@ -1,13 +1,13 @@
-#!/bin/bash
+#!/usr/bin/env bash

 set -e
 set -x

-if [ -f  ../../valkey/src/valkey-server ]; then
-    ../../valkey/src/valkey-server ./cache.conf
+if [ -f ../../valkey/src/valkey-server ]; then
+  ../../valkey/src/valkey-server ./cache.conf
 elif [ -f ../../redis/src/redis-server ]; then
-    ../../redis/src/redis-server ./cache.conf
+  ../../redis/src/redis-server ./cache.conf
 else
-    echo "Warning: using system redis-server. Valkey-server or redis-server from source is recommended." >&2
-    /usr/bin/redis-server ./cache.conf
+  echo "Warning: using system redis-server. Valkey-server or redis-server from source is recommended." >&2
+  redis-server ./cache.conf
 fi
diff --git a/indexing/run_redis.sh b/indexing/run_redis.sh
index d6924a0..7107500 100755
--- a/indexing/run_redis.sh
+++ b/indexing/run_redis.sh
@@ -1,13 +1,13 @@
-#!/bin/bash
+#!/usr/bin/env bash

 set -e
 set -x

-if [ -f  ../../valkey/src/valkey-server ]; then
-    ../../valkey/src/valkey-server ./indexing.conf
+if [ -f ../../valkey/src/valkey-server ]; then
+  ../../valkey/src/valkey-server ./indexing.conf
 elif [ -f ../../redis/src/redis-server ]; then
-    ../../redis/src/redis-server ./indexing.conf
+  ../../redis/src/redis-server ./indexing.conf
 else
-    echo "Warning: using system redis-server. Valkey-server or redis-server from source is recommended." >&2
-    /usr/bin/redis-server ./indexing.conf
+  echo "Warning: using system redis-server. Valkey-server or redis-server from source is recommended." >&2
+  redis-server ./indexing.conf
 fi
diff --git a/pyproject.toml b/pyproject.toml
index 7614ce8..2ded159 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -21,6 +21,11 @@ classifiers = [
     'Topic :: Internet',
 ]

+packages = [
+{include = "lookyloo"},
+{include = "bin"},
+]
+
 [tool.poetry.scripts]
 start = "bin.start:main"
 stop = "bin.stop:main"

The flake is :

{
  description = "Application packaged using poetry2nix";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    poetry2nix = {
      #url = "github:nix-community/poetry2nix";
      url = "git+https://gitea.com/oven_spinout988/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    flake-utils.lib.eachDefaultSystem
    (system:
      let
        pkgs = nixpkgs.legacyPackages.${system}.extend poetry2nix.overlays.default;

        projectDir = ./.;

        # NOTE: add non python packages, needed at runtime, here
        runtimePackages = with pkgs; [
          redis
          ffmpeg
          python3
        ];

        # NOTE: add non python packages, only needed for development, here
        devPackages = with pkgs; [
          pkgs.python3
          pkgs.poetry
        ];

        patches = [
          ./nix/cache.patch
          ./nix/indexing.patch
        ];

        pypkgs-build-requirements = {
          fsspec = [ "hatchling" "hatch-vcs" ];
          har2tree = [ "poetry-core" ];
          pydeep2 = [ "setuptools-scm" "ssdeep" pkgs.ssdeep pkgs.libtool pkgs.automake ];
          vt-py = [ "setuptools" "pytest-runner" ];
          pyeupi = [ "poetry-core" ];
          pyfaup = [ "setuptools" ];
          types-cffi = [ "setuptools" ];
          pypdns = [ "poetry-core" ];
          passivetotal = [ "setuptools" ];
          pyhashlookup = [ "poetry-core" ];
          pyipasnhistory = [ "poetry-core" ];
          pylookyloo = [ "poetry-core" ];
          pymisp = [ "poetry-core" ];
          pylookyloomonitoring = [ "poetry-core" ];
          pypandora = [ "poetry-core" ];
          pyphishtanklookup = [ "poetry-core" ];
          pysanejs = [ "poetry-core" ];
          pysecuritytxt = [ "poetry-core" ];
          pytaxonomies = [ "poetry-core" ];
          flask = [ "setuptools" ];
          ssdeep = [ "setuptools" ];
          lxml = [ pkgs.libz ];
          filetype = [ "setuptools" ];
        };
        p2n-overrides = pkgs.poetry2nix.overrides.withDefaults
          #p2n-overrides = defaultPoetryOverrides.extend
          (final: prev:
            builtins.mapAttrs
            (package: build-requirements:
              (builtins.getAttr package prev).overridePythonAttrs (old: {
                buildInputs = (old.buildInputs or [ ]) ++ (builtins.map (pkg: if builtins.isString pkg then builtins.getAttr pkg prev else pkg) build-requirements);
              })
            )

            pypkgs-build-requirements
          );

        propagatedBuildInputs = [
          pkgs.ffmpeg
          pkgs.redis
          # pkgs.python3
        ];

        lookylooApp = pkgs.poetry2nix.mkPoetryApplication
          {
            inherit projectDir;
            inherit propagatedBuildInputs;
            overrides = p2n-overrides;
          };
      in
        {
        packages = {
          default = lookylooApp;
        };

        apps = {
          start = {
            type = "app";
            program = "${lookylooApp}/bin/start";
          };

          start_website = {
            type = "app";
            program = "${lookylooApp}/bin/start_website";
          };

          stop = {
            type = "app";
            program = "${lookylooApp}/bin/stop";
          };
        };

        devShells.default =
          pkgs.mkShell
          {
            # inputsFrom = [ self.packages.${system}.lookyloo ];

            packages = [
              lookylooApp.dependencyEnv
            ]
            ++ runtimePackages
            ++ devPackages;

            shellHook = ''
                  export LOOKYLOO_HOME="$(pwd)";
                  export PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=true

                  echo "[+] Welcome to this nix shell"
                  echo "- Python $(python --version)"
                  '';
          };
      });
}

It is working with the dev environment.
Can you help me to find out what I am doing wrong ?