| 6 Nov 2023 |
 matthewcroughan | def tokenizer(self) -> Callable[[str], List]:
"""Get tokenizer."""
if self._tokenizer is None:
tiktoken_import_err = (
"`tiktoken` package not found, please run `pip install tiktoken`"
)
try:
import tiktoken
except ImportError:
raise ImportError(tiktoken_import_err)
enc = tiktoken.get_encoding("gpt2")
self._tokenizer = cast(Callable[[str], List], enc.encode)
self._tokenizer = partial(self._tokenizer, allowed_special="all")
return self._tokenizer # type: ignore
| 18:46:48 |
| matthewcroughan | seems reasonable | 18:46:50 |
 pareto-optimal-dev | Someone is adding something to importscheck here, not sure if related.. but interesting: https://sourcegraph.com/github.com/ibis-project/ibis/-/blob/nix/ibis.nix?L54 | 18:58:21 |
 K900 | pythonImportsCheck just checks the modules can be imported after the install | 19:00:02 |
| K900 | It doesn't actually affect what is installed | 19:00:12 |
| K900 | It's just a failsafe to make sure the package you've just built isn't completely broken | 19:00:23 |
| pareto-optimal-dev | Thanks for the clarification | 19:01:29 |
| pareto-optimal-dev | So they are just adding checks for each of the ibis backends | 19:01:42 |
| pareto-optimal-dev | Oh... I think this could be it, I'll have to try it when my build of your flake finishes. Adding python path to the resulting wrapper: https://sourcegraph.com/github.com/GTrunSec/dataflow2nix/-/blob/nix/prefect/packages/prefect.nix?L39 | 19:01:57 |
| pareto-optimal-dev | Interesting. nix run github:MatthewCroughan/privateGPT puts me in a python shell. Cloning the repo and doing nix run .# gives error: unable to execute '/nix/store/0c71l30nmfcmks194m1a16vryi3pjr34-python3.11-private-gpt-0.1.0/bin/private-gpt': No such file or directory. | 19:15:38 |
| matthewcroughan | that is odd | 19:16:02 |
| matthewcroughan | do --refresh on the nix run | 19:16:52 |
| pareto-optimal-dev | I get the error with both now | 19:17:24 |
| matthewcroughan | I force pushed a change, you must have ran the cmd before that change | 19:19:39 |
| matthewcroughan | previously I used mkPoetryEnv and now I'm using mkPoetryApplication | 19:19:48 |
| matthewcroughan | In reply to @pareto-optimal-dev:matrix.org
Oh... I think this could be it, I'll have to try it when my build of your flake finishes. Adding python path to the resulting wrapper: https://sourcegraph.com/github.com/GTrunSec/dataflow2nix/-/blob/nix/prefect/packages/prefect.nix?L39 Yeah but why isn't mkPoetryApplication doing this | 19:32:32 |
| matthewcroughan | even the mkPoetryEnv doesn't include tiktoken | 19:35:19 |
| matthewcroughan | ls result/lib/python3.11/site-packages/tiktoken-0.5.1.dist-info
license_files METADATA RECORD WHEEL
| 19:35:50 |
| matthewcroughan | it only has this in it | 19:35:52 |
| matthewcroughan | ah the install of tiktoken may not work.. | 19:36:09 |
| matthewcroughan | It definitely compiled, but maybe outPath has wrong stuff in it | 19:36:32 |
| matthewcroughan | python3.11-tiktoken> installPhase ()
python3.11-tiktoken> {
python3.11-tiktoken> runHook preInstall;
python3.11-tiktoken> if [[ -z "${makeFlags-}" && -z "${makefile:-}" && ! ( -e Makefile || -e makefile || -e GNUmakefile ) ]]; then
python3.11-tiktoken> echo "no Makefile or custom installPhase, doing nothing";
python3.11-tiktoken> runHook postInstall;
python3.11-tiktoken> return;
python3.11-tiktoken> else
python3.11-tiktoken> foundMakefile=1;
python3.11-tiktoken> fi;
python3.11-tiktoken> if [ -n "$prefix" ]; then
python3.11-tiktoken> mkdir -p "$prefix";
python3.11-tiktoken> fi;
python3.11-tiktoken> local flagsArray=(${enableParallelInstalling:+-j${NIX_BUILD_CORES}} SHELL=$SHELL);
python3.11-tiktoken> _accumFlagsArray makeFlags makeFlagsArray installFlags installFlagsArray;
python3.11-tiktoken> if [ -n "$__structuredAttrs" ]; then
python3.11-tiktoken> flagsArray+=("${installTargets[@]:-install}");
python3.11-tiktoken> else
python3.11-tiktoken> flagsArray+=(${installTargets:-install});
python3.11-tiktoken> fi;
python3.11-tiktoken> echoCmd 'install flags' "${flagsArray[@]}";
python3.11-tiktoken> make ${makefile:+-f $makefile} "${flagsArray[@]}";
python3.11-tiktoken> unset flagsArray;
python3.11-tiktoken> runHook postInstall
python3.11-tiktoken> }
| 19:37:35 |
| matthewcroughan | this is the installPhase for it | 19:37:38 |
| matthewcroughan | also
python3.11-tiktoken> ⚠ Warning: `build-backend` in pyproject.toml is not set to `maturin`, packaging tools such as pip will not use maturin to build this project.
| 19:38:02 |
| matthewcroughan | I included the maturin hook when it was not necessary | 19:39:41 |
| matthewcroughan | this lead to the failure, only included dist-info in the site-packages | 19:40:04 |
| matthewcroughan | pkgs.rustPlatform.cargoSetupHook
pkgs.rustc
pkgs.cargo
| 19:40:09 |
| matthewcroughan | instead, if I remove pkgs.rustPlatform.maturinBuildHook and just add rustc and cargo, it works | 19:40:24 |
| matthewcroughan | Ah good | 19:41:30 |
| matthewcroughan | python -m private_gpt
19:41:21.103 [INFO ] private_gpt.settings.settings_loader - Starting application with profiles=['default']
2023-11-06 19:41:21.194091237 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:1987 CreateInferencePybindStateModule] Init provider bridge failed.
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/home/matthew/git/privateGPT/private_gpt/__main__.py", line 5, in <module>
from private_gpt.main import app
File "/home/matthew/git/privateGPT/private_gpt/main.py", line 106, in <module>
from private_gpt.ui.ui import mount_in_app
File "/home/matthew/git/privateGPT/private_gpt/ui/ui.py", line 7, in <module>
import gradio as gr # type: ignore
^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'gradio
| 19:41:34 |
