| 9 Nov 2023 |
 matthewcroughan | I did manage to filter cmake out of the dependency graph by overriding 4 other python deps though, and just replacing it with pkgs.cmake instead of the python built one | 22:57:49 |
| matthewcroughan | This is what happens after I fix safetensors to be importable | 23:00:00 |
 K900 | In reply to @matthewcroughan:defenestrate.it
ls result/lib/python3.11/site-packages/tokenizers
tokenizers/ tokenizers_python-0.14.1.dist-info/
user: matthew 🌐 swordfish in privateGPT on main [!?⇡] is 📦 v0.1.0 via 🐍
✦ ❯ ls result/lib/python3.11/site-packages/tokenizers
__init__.py __pycache__ tokenizers.cpython-311-x86_64-linux-gnu.so
That module is literally not there | 23:01:38 |
| K900 | Why it's not there, I don't know | 23:01:43 |
| K900 | But it isn't | 23:01:45 |
| matthewcroughan | https://huggingface.co/docs/tokenizers/api/models | 23:03:57 |
| matthewcroughan | maybe because it's a wheel? | 23:05:10 |
| K900 | Wheels generally don't make modules disappear | 23:05:46 |
| matthewcroughan | https://github.com/huggingface/tokenizers/blob/648b33a09ee8da40a5b4599054eb855a7a875ca5/bindings/python/src/models.rs#L840 | 23:12:19 |
| matthewcroughan | looks like it's part of the rust code | 23:12:39 |
| matthewcroughan | During the build log for tokenizers it says
python3.11-tokenizers> ⚠️ Warning: You're building a library without activating pyo3's `extension-module` feature. See https://pyo3.rs/v0.19.2/building_and_distribution.html#linking
python3.11-tokenizers> 🔗 Found pyo3 bindings
python3.11-tokenizers> 🐍 Found CPython 3.11 at /nix/store/ffll6glz3gwx342z0ch8wx30p5cnqz1z-python3-3.11.5/bin/python3
python3.11-tokenizers> Compiling autocfg v1.1.
| 23:14:26 |
| K900 | https://github.com/huggingface/tokenizers/blob/648b33a09ee8da40a5b4599054eb855a7a875ca5/bindings/python/py_src/tokenizers/models/__init__.py | 23:20:49 |
| K900 | No it's not | 23:20:51 |
| matthewcroughan | the transformers from nixpkgs can do this import, that is good | 23:22:11 |
| matthewcroughan | K900 ⚡️: I literally copy pasted the derivation form nixpkgs, and made an override like this transformers = super.pythonPackages.callPackage ./f.nix {}; | 23:32:45 |
| matthewcroughan | is that dumb, or a good debugging step? | 23:33:02 |
| K900 | Could work | 23:33:11 |
| K900 | Could also not work | 23:33:14 |
| matthewcroughan | because in this case, the transformers can't import tokenizers.models and the one from nixpkgs can | 23:33:23 |
| matthewcroughan | they seem in the logs to build the same | 23:33:45 |
| 10 Nov 2023 |
| matthewcroughan | K900 ⚡️: the reason is because the version of transformers in the poetry.lock of privateGPT is 0.14.1 | 01:00:02 |
| matthewcroughan | whereas the version that had the method is 0.14.0, not 0.14.1 | 01:00:04 |
| matthewcroughan | they removed the method it's looking for in 0.14.1 | 01:04:20 |
| matthewcroughan | semantic versioning fail | 01:04:21 |
 adisbladis | This is really cool if I say so myself https://github.com/adisbladis/pdm2nix/tree/master/tests/nested-poetry
| 03:31:10 |
| adisbladis | a is a PDM project
b is a Poetry project | 03:31:20 |
