| 9 Nov 2023 |
 K900 | Python package names and module names are not actually required to match | 19:29:43 |
| K900 | As in, a package named safetensors_python can contain modules named safetensors, safetensors_rust or potato | 19:30:08 |
 matthewcroughan | import safetensors_rust
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/nix/store/fbdk8grc3nipn0z5prsfcmvd74pvppaf-python3-3.11.5-env/lib/python3.11/site-packages/safetensors_rust/__init__.py", line 1, in <module>
from .safetensors_rust import *
ImportError: dynamic module does not define module export function (PyInit_safetensors_rust)
| 19:39:54 |
| matthewcroughan | does this tell us anything? | 19:39:57 |
| matthewcroughan | I have no idea how the python binding stuff works | 19:40:28 |
| K900 | It probably tells you the library is very fucked | 19:42:19 |
| matthewcroughan | it seems to compile just fine, and the logs look very similar to transformers which does work correctly | 19:42:34 |
| matthewcroughan | * it seems to compile just fine, and the logs look very similar to transformers which does work correctly and builds in the same way | 19:42:38 |
| matthewcroughan | https://github.com/nixified-ai/flake/blob/master/packages/safetensors/default.nix | 19:43:13 |
| matthewcroughan | maybe the major version has something to do with it | 19:43:37 |
| matthewcroughan | nixpkgs, and nixified ai has 0.3.3, whereas what's being asked for is 0.4.0, maybe they did something | 19:43:53 |
| matthewcroughan | import transformers.models.auto
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/nix/store/qqlksqz6x91k9w4pn8mj2dc2s077zbmd-python3.11-transformers-4.34.0/lib/python3.11/site-packages/transformers/models/__init__.py", line 15, in <module>
from . import (
File "/nix/store/qqlksqz6x91k9w4pn8mj2dc2s077zbmd-python3.11-transformers-4.34.0/lib/python3.11/site-packages/transformers/models/mt5/__init__.py", line 29, in <module>
from ..t5.tokenization_t5 import T5Tokenizer
File "/nix/store/qqlksqz6x91k9w4pn8mj2dc2s077zbmd-python3.11-transformers-4.34.0/lib/python3.11/site-packages/transformers/models/t5/tokenization_t5.py", line 26, in <module>
from ...convert_slow_tokenizer import import_protobuf
File "/nix/store/qqlksqz6x91k9w4pn8mj2dc2s077zbmd-python3.11-transformers-4.34.0/lib/python3.11/site-packages/transformers/convert_slow_tokenizer.py", line 27, in <module>
from tokenizers.models import BPE, Unigram, WordPiece
ModuleNotFoundError: No module named 'tokenizers.models'
| 22:56:04 |
| matthewcroughan | I have not been able to debug why this happens either. | 22:56:32 |
| matthewcroughan | I did manage to filter cmake out of the dependency graph by overriding 4 other python deps though, and just replacing it with pkgs.cmake instead of the python built one | 22:57:49 |
| matthewcroughan | This is what happens after I fix safetensors to be importable | 23:00:00 |
| K900 | In reply to @matthewcroughan:defenestrate.it
ls result/lib/python3.11/site-packages/tokenizers
tokenizers/ tokenizers_python-0.14.1.dist-info/
user: matthew 🌐 swordfish in privateGPT on main [!?⇡] is 📦 v0.1.0 via 🐍
✦ ❯ ls result/lib/python3.11/site-packages/tokenizers
__init__.py __pycache__ tokenizers.cpython-311-x86_64-linux-gnu.so
That module is literally not there | 23:01:38 |
| K900 | Why it's not there, I don't know | 23:01:43 |
| K900 | But it isn't | 23:01:45 |
| matthewcroughan | https://huggingface.co/docs/tokenizers/api/models | 23:03:57 |
| matthewcroughan | maybe because it's a wheel? | 23:05:10 |
| K900 | Wheels generally don't make modules disappear | 23:05:46 |
| matthewcroughan | https://github.com/huggingface/tokenizers/blob/648b33a09ee8da40a5b4599054eb855a7a875ca5/bindings/python/src/models.rs#L840 | 23:12:19 |
| matthewcroughan | looks like it's part of the rust code | 23:12:39 |
| matthewcroughan | During the build log for tokenizers it says
python3.11-tokenizers> ⚠️ Warning: You're building a library without activating pyo3's `extension-module` feature. See https://pyo3.rs/v0.19.2/building_and_distribution.html#linking
python3.11-tokenizers> 🔗 Found pyo3 bindings
python3.11-tokenizers> 🐍 Found CPython 3.11 at /nix/store/ffll6glz3gwx342z0ch8wx30p5cnqz1z-python3-3.11.5/bin/python3
python3.11-tokenizers> Compiling autocfg v1.1.
| 23:14:26 |
| K900 | https://github.com/huggingface/tokenizers/blob/648b33a09ee8da40a5b4599054eb855a7a875ca5/bindings/python/py_src/tokenizers/models/__init__.py | 23:20:49 |
| K900 | No it's not | 23:20:51 |
| matthewcroughan | the transformers from nixpkgs can do this import, that is good | 23:22:11 |
| matthewcroughan | K900 ⚡️: I literally copy pasted the derivation form nixpkgs, and made an override like this transformers = super.pythonPackages.callPackage ./f.nix {}; | 23:32:45 |
| matthewcroughan | is that dumb, or a good debugging step? | 23:33:02 |
| K900 | Could work | 23:33:11 |
