| 29 Jan 2022 |
mou | In reply to @rolandco:matrix.org To reproduce, nix develop; graphtage --help pyproject.toml and poetry.lock needed too | 16:54:24 |
mou | or just pyproject.toml | 16:54:59 |
Roland Coeurjoly | In reply to @mou_bugtracker:matrix.org or just pyproject.toml [tool.poetry] name = “graphtage-tool” version = “0.1.0” description = “” authors = [“Roland Coeurjoly rolandcoeurjoly@gmail.com”]
[tool.poetry.dependencies] python = “>=3.9,<3.11” graphtage = “^0.2.5” wildq = “^1.1.10”
[tool.poetry.dev-dependencies]
[build-system] requires = [“poetry-core>=1.0.0”] build-backend = “poetry.core.masonry.api” | 17:00:06 |
Roland Coeurjoly | Sorry | 17:00:34 |
Roland Coeurjoly | How can you reproduce without poetry.lock? Nix complaints without it | 17:01:26 |
Roland Coeurjoly | https://github.com/RCoeurjoly/nix_graphtage | 17:11:45 |
Roland Coeurjoly | mou: you can find the repo above | 17:12:17 |
mou | thanks | 17:12:36 |
mou | it take time to build | 17:12:44 |
Roland Coeurjoly | The package graphtage is in nixpkgs, and I can create a shell with it easily | 18:52:16 |
Roland Coeurjoly | But wildq is not in nixpkgs | 18:52:29 |
Roland Coeurjoly | Is there a way to create a shell with python modules from nixpkgs and also some installed with poetry? | 18:54:10 |
mou | unfortunately my time is over for today. I was not able to figure out cause of problem. But i think cause lies in missing native build dependencies like gfortran and similar. Here is definition for scypi in nixpkgs https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/scipy/default.nix
And | 19:36:21 |
mou | Consider use poetry2nix not from nixpkgs, but latest version from git. because it has pretty haevy override for scipy https://github.com/nix-community/poetry2nix/blob/master/overrides/default.nix#L1671 | 19:39:18 |
mou | i take standard template for poetry2nix and tried to build. ended up with cythonization fail ((
Processing scipy/optimize/_group_columns.pyx
Processing scipy/interpolate/interpnd.pyx
Processing scipy/optimize/_highs/cython/src/_highs_wrapper.pyx
Processing scipy/optimize/_highs/cython/src/_highs_constants.pyx
Traceback (most recent call last):
File "/build/scipy-1.7.3/tools/cythonize.py", line 324, in <module>
main()
File "/build/scipy-1.7.3/tools/cythonize.py", line 320, in main
find_process_files(root_dir)
File "/build/scipy-1.7.3/tools/cythonize.py", line 309, in find_process_files
Processing scipy/optimize/_trlib/_trlib.pyx
for result in pool.imap_unordered(lambda args: process(*args), jobs):
File "/nix/store/i6vabb4div9iy6lsl642d86k1q8riasn-python3-3.9.9/lib/python3.9/multiprocessing/pool.py", line 870, in next
raise value
File "/nix/store/i6vabb4div9iy6lsl642d86k1q8riasn-python3-3.9.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/build/scipy-1.7.3/tools/cythonize.py", line 309, in <lambda>
for result in pool.imap_unordered(lambda args: process(*args), jobs):
File "/build/scipy-1.7.3/tools/cythonize.py", line 243, in process
processor_function(fromfile, tofile, cwd=path)
File "/build/scipy-1.7.3/tools/cythonize.py", line 80, in process_pyx
min_required_version, max_required_version = line.split(split_on)
ValueError: too many values to unpack (expected 2)
Processing scipy/optimize/cython_optimize/_zeros.pyx.in
Cythonizing sources
| 19:54:23 |
mou | * i took standard template for poetry2nix and tried to build. ended up with cythonization fail ((
Processing scipy/optimize/_group_columns.pyx
Processing scipy/interpolate/interpnd.pyx
Processing scipy/optimize/_highs/cython/src/_highs_wrapper.pyx
Processing scipy/optimize/_highs/cython/src/_highs_constants.pyx
Traceback (most recent call last):
File "/build/scipy-1.7.3/tools/cythonize.py", line 324, in <module>
main()
File "/build/scipy-1.7.3/tools/cythonize.py", line 320, in main
find_process_files(root_dir)
File "/build/scipy-1.7.3/tools/cythonize.py", line 309, in find_process_files
Processing scipy/optimize/_trlib/_trlib.pyx
for result in pool.imap_unordered(lambda args: process(*args), jobs):
File "/nix/store/i6vabb4div9iy6lsl642d86k1q8riasn-python3-3.9.9/lib/python3.9/multiprocessing/pool.py", line 870, in next
raise value
File "/nix/store/i6vabb4div9iy6lsl642d86k1q8riasn-python3-3.9.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/build/scipy-1.7.3/tools/cythonize.py", line 309, in <lambda>
for result in pool.imap_unordered(lambda args: process(*args), jobs):
File "/build/scipy-1.7.3/tools/cythonize.py", line 243, in process
processor_function(fromfile, tofile, cwd=path)
File "/build/scipy-1.7.3/tools/cythonize.py", line 80, in process_pyx
min_required_version, max_required_version = line.split(split_on)
ValueError: too many values to unpack (expected 2)
Processing scipy/optimize/cython_optimize/_zeros.pyx.in
Cythonizing sources
| 21:53:16 |
| 3 Feb 2022 |
| jyamad joined the room. | 20:43:40 |
| lewo joined the room. | 21:47:31 |
| 8 Feb 2022 |
| Dominic Egger joined the room. | 05:38:54 |
Dominic Egger | Good Morning everyone. I am currently working on a little showcase workbench/projects for python based friends that don't know nix. They work fairly extensively with numpy and pandas so i tried adding those to poetry and so also to my environment. but it seems that that causes pandas and numpy to fully recompile them locally. i also tried pinning the versions to ones specifically available on nix. does anyone have some insight into this? | 05:41:33 |
adisbladis | In reply to @graf_blutwurst:matrix.org Good Morning everyone. I am currently working on a little showcase workbench/projects for python based friends that don't know nix. They work fairly extensively with numpy and pandas so i tried adding those to poetry and so also to my environment. but it seems that that causes pandas and numpy to fully recompile them locally. i also tried pinning the versions to ones specifically available on nix. does anyone have some insight into this? That's fully expected. Even if you would somehow pin the exact same versions that are available in nixpkgs the build closure and resulting derivations are not exactly the same. | 05:52:31 |
Dominic Egger | hm it seems i have to better understand build closures better. i assumed a dependency like that would be "low" enough that it could be picked up from the store/nixpkgs. thanks though! how do people deal with the long resulting build times? push the result from mkPoetryEnv to a binary cache? | 06:02:07 |
adisbladis | In reply to @graf_blutwurst:matrix.org hm it seems i have to better understand build closures better. i assumed a dependency like that would be "low" enough that it could be picked up from the store/nixpkgs. thanks though! how do people deal with the long resulting build times? push the result from mkPoetryEnv to a binary cache? Typically, yes. | 06:02:46 |
Dominic Egger | In reply to @adis:blad.is Typically, yes. ok then i was on the right track for that. do you happen to have any resources that'd help me understand what causes the recompile of pandas etc? mostly so i can predict when it happens since those take a significant chunk of time to crunch through | 06:06:31 |
adisbladis | In reply to @graf_blutwurst:matrix.org ok then i was on the right track for that. do you happen to have any resources that'd help me understand what causes the recompile of pandas etc? mostly so i can predict when it happens since those take a significant chunk of time to crunch through When some of the build inputs changes, or some parameters are different.
poetry2nix tries as much as possible to ignore nixpkgs python packages which will cause the derivations to be different from nixpkgs ~all of the time.
Practially this means that you will have to rebuild when:
- You update nixpkgs
- Any of the python packages that pandas/numpy depends on changes in your poetry.lock
| 06:08:37 |
Dominic Egger | that sounds sensible.thanks for your time! | 06:17:11 |
| 12 Feb 2022 |
| Octavio Martín Duarte joined the room. | 15:26:44 |
Octavio Martín Duarte | Hello! I'm suffering from a problem that seems to have been already detected and fixed...
I get this failure even for the Flask exaple from the Tweag blogpost and the video. I get the message about an infinite recursion and when I use the verbose switch it get to read that flint-core is detected as missing. Adding the package as a dependendency does not stop this from failing.
| 15:54:06 |
mou | Octavio, could you provide details about your environment? Error message, pyproject.toml, your nix expression will help anyone to better identify root cause. | 18:45:31 |
| 14 Feb 2022 |
Octavio Martín Duarte | Download pyproject.toml | 02:09:18 |