| 6 Nov 2023 |
 pareto-optimal-dev | tiktokens for not committing their lockfile | 18:00:40 |
 matthewcroughan - nix.how | Okay, so they have not committed a lockfile, that means you cannot build their software with Nix. | 18:00:57 |
| matthewcroughan - nix.how | If you want that to happen, you need to use another build system that will generate different results every time you try to build the software. | 18:01:42 |
| matthewcroughan - nix.how | Or get the lockfile, and pass it to Nix, yourself. | 18:02:12 |
| matthewcroughan - nix.how | So we can see here, that there is absolutely nothing wrong with Nix. Instead, the problem is with tiktoken. | 18:02:30 |
| matthewcroughan - nix.how | This problem is present in most Python software. | 18:02:39 |
| matthewcroughan - nix.how | * This problem is present in most Python software too. The lack of a lockfile. | 18:02:47 |
| matthewcroughan - nix.how | Or an incomplete lockfile. | 18:03:09 |
| matthewcroughan - nix.how | So what does pip do? It ignores that and gives different results when you run it twice. | 18:03:28 |
| pareto-optimal-dev | Yes. The problem is "figure out workaround Nix needs because they don't value reproducibility" is it can be costly in terms of time and lots of times different per language. I'd prefer to always both get Nix to work and push the upstream to use a lock file. | 18:04:14 |
| matthewcroughan - nix.how | Yes, so why don't you just not use their software? | 18:04:25 |
| pareto-optimal-dev | Because it offers value. For instance MemGPT can help me with remembering those Nix overrides, perhaps better than my current notes ;) | 18:05:10 |
| matthewcroughan - nix.how | There's a lot of good Rust stuff coming out, regarding LLMs and machine learning in general | 18:05:12 |
| matthewcroughan - nix.how | poetry2nix, and other projects like it, are Nix code that scrambles to find correct input data, to automatically build python packages. | 18:05:36 |
| matthewcroughan - nix.how | but Python has seen an explosion in build systems, complexity, making it deeply unautomatable. | 18:05:50 |
| matthewcroughan - nix.how | poetry2nix still reduces the amount of code required to do something, massively | 18:05:59 |
| matthewcroughan - nix.how | * poetry2nix still reduces the amount of code required to build something, massively | 18:06:05 |
| matthewcroughan - nix.how | But in this case, all you have to do is generate a Cargo.lock, and give it to Nix. | 18:06:28 |
| matthewcroughan - nix.how | When you use pip, etc, you're just delaying the inevitable undebuggable, boring, uninteresting pipeline issues that only exist because people aren't defining their dependencies clearly | 18:07:20 |
| matthewcroughan - nix.how | and when you run into such an issue, you'll just have to wipe it and start over, because you can't share that with anyone, the issue would be unique to your personal environment. | 18:07:42 |
| pareto-optimal-dev | In reply to @matthewcroughan:defenestrate.it
But in this case, all you have to do is generate a Cargo.lock, and give it to Nix. Which isn't too hard, and is manageable to figure out... the issue comes when it is "how do I generate a cargo.lock and tell poetry2nix about it". At least it's those kinds of integration issues that usually take time for me. | 18:08:33 |
| matthewcroughan - nix.how | So the difference is being able to reproduce something because it was defined correctly in the first place, vs pure experimentation and unreproducible environments that you cannot share with anyone | 18:09:14 |
| matthewcroughan - nix.how | In reply to @pareto-optimal-dev:matrix.org
Which isn't too hard, and is manageable to figure out... the issue comes when it is "how do I generate a cargo.lock and tell poetry2nix about it". At least it's those kinds of integration issues that usually take time for me. Not hard, just have to learn Nix | 18:09:29 |
| matthewcroughan - nix.how | And also the build system for Rust. | 18:09:42 |
| matthewcroughan - nix.how | If you don't like that, you need to complain upstream and ask why they used Rust and Python together. | 18:09:56 |
| matthewcroughan - nix.how | Alternatively write your own code, or stop using their code. | 18:10:07 |
| matthewcroughan - nix.how | Do you see now how this has nothing to do with Nix, just software in general? | 18:10:29 |
| pareto-optimal-dev | In reply to @matthewcroughan:defenestrate.it
Not hard, just have to learn Nix Well, I know Nix the language, and I know how to use Nix for Haskell... it's just hard to transfer that knowledge. I guess though learning the build system is a big part of the complexity too. | 18:10:46 |
| matthewcroughan - nix.how | Some software has fundamental complexity that will not be resolved or masked by any tool if you actually want to reproduce it | 18:11:38 |
| matthewcroughan - nix.how | The only alternative to reproducing something, is to give someone something you built, an artifact, and that's not correct. | 18:11:51 |
