| 24 Jun 2026 |
 Vladimír Čunát | If a builder needs a path which is missing in S3, it will build and upload it, if I understand the current implementation right. | 07:11:46 |
| Vladimír Čunát | That helps recovery. But we have some leaf builds left which show succeeded on Hydra but don't have a (usable) result in S3. | 07:12:52 |
 K900 | Not sure why, but the build logs no longer strip ANSI escapes | 07:18:14 |
| K900 | Might have something to do with how the new runner calls the builds? | 07:18:25 |
| Vladimír Čunát | Tracked in https://github.com/NixOS/hydra/issues/1802 | 07:20:52 |
| K900 | Honestly rendering on the frontend should be pretty doable with something like https://github.com/drudru/ansi_up | 07:24:07 |
 dramforever | speaking of, i just realized that the build logs doesn't seem to be uploaded to cache.nixos.org anymore. is this normal? | 07:24:08 |
| dramforever | * speaking of logs, i just realized that the build logs doesn't seem to be uploaded to cache.nixos.org anymore. is this normal? | 07:24:13 |
| K900 | (I think that one is what gitlab CI is using) | 07:24:16 |
 Mic92 | Looks like clients are racing to upload nars, which wouldn't be a problem for nix but than fastly caches different filesizes because of non-reproducible compression, which produces the error we see. | 07:26:17 |
| dramforever | for example, this is a build https://hydra.nixos.org/build/331688229/nixlog/160 for /nix/store/wvhfhvkgb4qwlhl5d32sjrr9bw13g02b-nixos-graphical-26.11pre1019198.459bc06669f8-x86_64-linux.iso.drv, with output /nix/store/9rgmsc5vxymhmj2jf8rmz3m52f49j038-nixos-graphical-26.11pre1019198.459bc06669f8-x86_64-linux.iso. The latter is on c.n.o, but https://cache.nixos.org/log/wvhfhvkgb4qwlhl5d32sjrr9bw13g02b-nixos-graphical-26.11pre1019198.459bc06669f8-x86_64-linux.iso.drv is 404 | 07:26:22 |
| Mic92 | To fix the existing errors, we can flush the fastly cache. | 07:27:13 |
| Vladimír Čunát | That sounds like a design problem? We need to lock each .drv hash in the queue-runner then? | 07:27:49 |
| Mic92 | and the hydra fix would be s3 upload + If-None-Match | 07:27:59 |
| Mic92 | so it's write once | 07:28:07 |
| Mic92 | The looser will try to download the nar as well in case the original uploader never returns | 07:28:27 |
| K900 | OK actually it's like five lines of code | 07:28:59 |
| K900 | Let me just do a PR | 07:29:01 |
| Vladimír Čunát | I still wouldn't be too happy if a builder built stuff against a local version of a dependency instead of waiting for the official one which ends up in S3. | 07:29:29 |
| Vladimír Čunát | * I still wouldn't be too happy if a builder built stuff against a local version of a dependency instead of waiting for the "official one" which ends up in S3. | 07:29:40 |
| Vladimír Čunát | Smells like rare weird effects if for stuff which isn't 100% binary reproducible. | 07:30:15 |
| Vladimír Čunát | * Smells like rare weird effects for stuff which isn't 100% binary reproducible. | 07:30:22 |
| Mic92 | it's actually the same NAR in this case | 07:30:29 |
| Mic92 | so same content | 07:30:43 |
| Vladimír Čunát | I meant generally. | 07:30:57 |
| Vladimír Čunát | We probably can't afford to assume that the NAR would always be the same. | 07:31:22 |
| Mic92 | I think queue runner itself should dedup derivation already at a higher level | 07:32:10 |
| Mic92 | so it's not like the same build run twice | 07:32:37 |
| dramforever | we upload nar/{narhash}.nar.zst now right? | 07:32:48 |
| Vladimír Čunát | 🤔 but that won't persist if we restart the queue-runner and the leftover build result might persist on the builder? | 07:32:54 |
