| 6 Nov 2023 |
 matthewcroughan | tiktoken = super.tiktoken.overridePythonAttrs
(
old: {
postPatch = ''
ln -s ${./nix/tiktoken/Cargo.lock} Cargo.lock
'';
cargoDeps = pkgs.rustPlatform.importCargoLock {
lockFile = ./nix/tiktoken/Cargo.lock;
};
nativeBuildInputs = (old.nativeBuildInputs or [ ]) ++ [
pkgs.rustPlatform.cargoSetupHook
pkgs.rustPlatform.maturinBuildHook
];
}
);
| 18:16:23 |
| matthewcroughan | poetry2nix cannot just package things of this complexity for you, yet | 18:16:41 |
| matthewcroughan | suddenly, people start using rust and python together, and they start making new tools to deal with that and build that, it cannot be expected of poetry2nix to automatically deal with this new emerging class of usage | 18:17:28 |
| matthewcroughan | Is there even enough input information in the src to do it? | 18:17:56 |
| matthewcroughan | Another annoying thing seems to be that the Cargo.lock is available if you get it from pypi, but not from github | 18:19:39 |
 pareto-optimal-dev | In reply to @matthewcroughan:defenestrate.it
Another annoying thing seems to be that the Cargo.lock is available if you get it from pypi, but not from github In the issue above they were less resistant to adding it to pypi releases. | 18:20:09 |
| pareto-optimal-dev | In reply to @matthewcroughan:defenestrate.it
tiktoken = super.tiktoken.overridePythonAttrs
(
old: {
postPatch = ''
ln -s ${./nix/tiktoken/Cargo.lock} Cargo.lock
'';
cargoDeps = pkgs.rustPlatform.importCargoLock {
lockFile = ./nix/tiktoken/Cargo.lock;
};
nativeBuildInputs = (old.nativeBuildInputs or [ ]) ++ [
pkgs.rustPlatform.cargoSetupHook
pkgs.rustPlatform.maturinBuildHook
];
}
);
Do you generate /nix/tiktoken/Cargo.lock yourself? | 18:21:07 |
| matthewcroughan | yes | 18:21:11 |
| matthewcroughan | you go to the source code, you type cargo generate-lockfile, then you have the lock file. | 18:21:23 |
| matthewcroughan | every single time they update the library, you will have to do that. | 18:21:35 |
| matthewcroughan | If you have 5 dependencies that have this problem, you will have to do it 5 times, and you will have to check it in yourself | 18:21:50 |
| matthewcroughan | We chose Nix, not because it is easy, but because it is hard. | 18:22:49 |
| matthewcroughan | Although, again, Nix won't be the only tool that has problems dealing with the lack of specification in Python. | 18:23:28 |
| pareto-optimal-dev | This looks interesting for the cargo problem, but then I'll have to learn the flake parts stuff it seems :D
https://github.com/yusdacra/nix-cargo-integration | 18:23:50 |
| matthewcroughan | It is just information. Is there enough information in the repo to reproduce something/ | 18:23:55 |
| matthewcroughan | * It is just information. Is there enough information in the repo to reproduce something? | 18:23:56 |
| matthewcroughan | If there is not enough information in the source code, then even a human will have issues discerning how to reproduce what is inside of it. | 18:24:11 |
| matthewcroughan | It's like OCR, it's error prone because of lack of accuracy | 18:24:40 |
| matthewcroughan | I have an issue now with llama-index :( | 18:30:20 |
| matthewcroughan | llama-index = super.llama-index.overridePythonAttrs
(
old: {
propagatedBuildInputs = (old.propagatedBuildInputs or [ ]) ++ [ self.tiktoken ];
nativeBuildInputs = (old.nativeBuildInputs or [ ]) ++ [ super.poetry ];
}
);
| 18:30:22 |
| matthewcroughan | Traceback (most recent call last):
File "/nix/store/3g04wving6mlz1vxmnaqxy1wfsyh8g50-python3.11-llama-index-0.8.47/lib/python3.11/site-packages/llama_index/utils.py", line 49, in tokenizer
import tiktoken
ModuleNotFoundError: No module named 'tiktoken'
| 18:31:00 |
| pareto-optimal-dev | It has poetry.lock at least :) | 18:31:13 |
| matthewcroughan | https://github.com/imartinez/privateGPT | 18:31:54 |
| pareto-optimal-dev | is it because it wants version 0.3.3? | 18:31:54 |
| matthewcroughan | I'm building this | 18:31:57 |
| pareto-optimal-dev | I packaged a similar one, khoj: https://khoj.dev/
I want to compare them though. | 18:32:38 |
| pareto-optimal-dev | Oh privateGPT has a RAG/memory similar to memgpt and khoj doesn't IIRC. | 18:33:34 |
| matthewcroughan | In reply to @pareto-optimal-dev:matrix.org
is it because it wants version 0.3.3? maybe, but it should have killed the build if it didn't have the version | 18:34:10 |
| matthewcroughan | llama-index builds fine, it just doesn't run fine | 18:34:17 |
| pareto-optimal-dev | so... the python packages directory it's being run with doesn't contain tiktoken for some reason? | 18:34:50 |
