| 6 Nov 2023 |
 matthewcroughan | yeah, it has taken me hours to do this stuff too, where docker would have potentially taken a few minutes | 17:50:38 |
| matthewcroughan | but the missing information in that statement, is that Docker would have only worked for a few minutes | 17:50:49 |
| matthewcroughan | Once done in Nix, it will continue working, the Dockerfile for the project is probably not reproducible. | 17:51:26 |
| matthewcroughan | So if you care about reproducibility, you kind of have to do it in Nix | 17:51:36 |
 pareto-optimal-dev | In reply to @matthewcroughan:defenestrate.it
So if you care about reproducibility, you kind of have to do it in Nix My problem is typically putting too much time/effort into packaging with Nix honestly. | 17:58:03 |
| matthewcroughan | It's not Nix. It's software in general. | 17:58:29 |
| pareto-optimal-dev | Also, here is an issue related to poetry2nix and tiktoken, not sure if you've seen it:
https://github.com/nix-community/poetry2nix/issues/1155 | 17:58:22 |
| matthewcroughan | I see a lot of misattributed blame on Nix. | 17:58:44 |
| pareto-optimal-dev | I agree that Nix surfaces the mess that crappy build practices have made. However, sometimes I don't have enough free time to counteract that by getting Nix to work ;) | 17:59:23 |
| matthewcroughan | Okay, we can debug this. | 17:59:38 |
| matthewcroughan | First,
Validating consistency between /build/Cargo.lock and /Cargo.lock
/nix/store/546ykj3xf8w5vznnk4m29a0wkzi4phf7-diffutils-3.9/bin/diff: /Cargo.lock: No such file or directory
| 18:00:00 |
| matthewcroughan | Now, you tell me, who's fault is that? | 18:00:05 |
| matthewcroughan | Is that nix's fault? Or is that tiktoken's fault?. | 18:00:12 |
| matthewcroughan | The fact that there is no Cargo.lock in the tiktoken repository, leading to unreproducibility. | 18:00:29 |
| pareto-optimal-dev | tiktokens for not committing their lockfile | 18:00:40 |
| matthewcroughan | Okay, so they have not committed a lockfile, that means you cannot build their software with Nix. | 18:00:57 |
| matthewcroughan | If you want that to happen, you need to use another build system that will generate different results every time you try to build the software. | 18:01:42 |
| matthewcroughan | Or get the lockfile, and pass it to Nix, yourself. | 18:02:12 |
| matthewcroughan | So we can see here, that there is absolutely nothing wrong with Nix. Instead, the problem is with tiktoken. | 18:02:30 |
| matthewcroughan | This problem is present in most Python software. | 18:02:39 |
| matthewcroughan | * This problem is present in most Python software too. The lack of a lockfile. | 18:02:47 |
| matthewcroughan | Or an incomplete lockfile. | 18:03:09 |
| matthewcroughan | So what does pip do? It ignores that and gives different results when you run it twice. | 18:03:28 |
| pareto-optimal-dev | Yes. The problem is "figure out workaround Nix needs because they don't value reproducibility" is it can be costly in terms of time and lots of times different per language. I'd prefer to always both get Nix to work and push the upstream to use a lock file. | 18:04:14 |
| matthewcroughan | Yes, so why don't you just not use their software? | 18:04:25 |
| pareto-optimal-dev | Because it offers value. For instance MemGPT can help me with remembering those Nix overrides, perhaps better than my current notes ;) | 18:05:10 |
| matthewcroughan | There's a lot of good Rust stuff coming out, regarding LLMs and machine learning in general | 18:05:12 |
| matthewcroughan | poetry2nix, and other projects like it, are Nix code that scrambles to find correct input data, to automatically build python packages. | 18:05:36 |
| matthewcroughan | but Python has seen an explosion in build systems, complexity, making it deeply unautomatable. | 18:05:50 |
| matthewcroughan | poetry2nix still reduces the amount of code required to do something, massively | 18:05:59 |
