| 6 Nov 2023 |
 matthewcroughan | because builtins.fetchTarball isn't being used, so src refers to the tar.gz, not the outPath (narHashed output) | 17:46:23 |
 pareto-optimal-dev | Hm... I guess in the short term I'll have to go back to trying to get docker to work with gpu's then it sounds like. | 17:47:26 |
| matthewcroughan | No, you just have to learn Nix, and do the hard work | 17:48:02 |
| matthewcroughan | if you want it to work for a day, use docker, if you want to use it for a year, or maybe build something that will last, use nix | 17:48:14 |
| matthewcroughan | I know that's tough, but it is true in the python ecosystem | 17:48:30 |
| pareto-optimal-dev | Yeah, it's very brittle. I've been using Nix for many years and manage my system with it. Which is why I know "you just have to learn Nix and do the hardwork" is true but in cases like this could take 100+ hours for me. | 17:49:23 |
| matthewcroughan | yeah, it has taken me hours to do this stuff too, where docker would have potentially taken a few minutes | 17:50:38 |
| matthewcroughan | but the missing information in that statement, is that Docker would have only worked for a few minutes | 17:50:49 |
| matthewcroughan | Once done in Nix, it will continue working, the Dockerfile for the project is probably not reproducible. | 17:51:26 |
| matthewcroughan | So if you care about reproducibility, you kind of have to do it in Nix | 17:51:36 |
| pareto-optimal-dev | In reply to @matthewcroughan:defenestrate.it
So if you care about reproducibility, you kind of have to do it in Nix My problem is typically putting too much time/effort into packaging with Nix honestly. | 17:58:03 |
| matthewcroughan | It's not Nix. It's software in general. | 17:58:29 |
| pareto-optimal-dev | Also, here is an issue related to poetry2nix and tiktoken, not sure if you've seen it:
https://github.com/nix-community/poetry2nix/issues/1155 | 17:58:22 |
| matthewcroughan | I see a lot of misattributed blame on Nix. | 17:58:44 |
| pareto-optimal-dev | I agree that Nix surfaces the mess that crappy build practices have made. However, sometimes I don't have enough free time to counteract that by getting Nix to work ;) | 17:59:23 |
| matthewcroughan | Okay, we can debug this. | 17:59:38 |
| matthewcroughan | First,
Validating consistency between /build/Cargo.lock and /Cargo.lock
/nix/store/546ykj3xf8w5vznnk4m29a0wkzi4phf7-diffutils-3.9/bin/diff: /Cargo.lock: No such file or directory
| 18:00:00 |
| matthewcroughan | Now, you tell me, who's fault is that? | 18:00:05 |
| matthewcroughan | Is that nix's fault? Or is that tiktoken's fault?. | 18:00:12 |
| matthewcroughan | The fact that there is no Cargo.lock in the tiktoken repository, leading to unreproducibility. | 18:00:29 |
| pareto-optimal-dev | tiktokens for not committing their lockfile | 18:00:40 |
| matthewcroughan | Okay, so they have not committed a lockfile, that means you cannot build their software with Nix. | 18:00:57 |
| matthewcroughan | If you want that to happen, you need to use another build system that will generate different results every time you try to build the software. | 18:01:42 |
| matthewcroughan | Or get the lockfile, and pass it to Nix, yourself. | 18:02:12 |
| matthewcroughan | So we can see here, that there is absolutely nothing wrong with Nix. Instead, the problem is with tiktoken. | 18:02:30 |
| matthewcroughan | This problem is present in most Python software. | 18:02:39 |
| matthewcroughan | * This problem is present in most Python software too. The lack of a lockfile. | 18:02:47 |
| matthewcroughan | Or an incomplete lockfile. | 18:03:09 |
| matthewcroughan | So what does pip do? It ignores that and gives different results when you run it twice. | 18:03:28 |
| pareto-optimal-dev | Yes. The problem is "figure out workaround Nix needs because they don't value reproducibility" is it can be costly in terms of time and lots of times different per language. I'd prefer to always both get Nix to work and push the upstream to use a lock file. | 18:04:14 |
