| 24 Jun 2026 |
 Vladimír Čunát | *
error: failed to read compressed data (Zstd decompression failed: Data corruption detected) error: path /nix/store/xbpvckqw1v2bg2p1r6jl6s3viny9rjiv-clang-src-22.1.8 is required, but there is no substituter that can build it
We have these failures in several packages, blocked at least a thousand jobs.
| 05:50:58 |
| Vladimír Čunát | When I fetch the path now locally (nix build /nix/store...), it succeeds. | 05:52:43 |
| Vladimír Čunát | But I can reproduce this one
$ nix build /nix/store/pbgwd0c61p5p0c1fmklx30wqk7q33nj5-rusty-v8-149.3.0-vendor
warning: error: unable to download 'https://cache.nixos.org/nar/1imr6zrqbb8l9l933yhsxsxl7j0ygs4prrrn50aiicr3hxp23n0w.nar.zst': HTTP error 200 (curl error: Stream error in the HTTP/2 framing layer); retrying from offset 8388608 in 253 ms
error: failed to read compressed data (Zstd decompression failed: Data corruption detected)
error: path '/nix/store/pbgwd0c61p5p0c1fmklx30wqk7q33nj5-rusty-v8-149.3.0-vendor' is required, but there is no substituter that can build it
| 05:53:57 |
|  c2fc2f joined the room. | 06:48:51 |
 Sergei Zimmerman (xokdvium) | Huh, doing curl https://cache.nixos.org/nar/1imr6zrqbb8l9l933yhsxsxl7j0ygs4prrrn50aiicr3hxp23n0w.nar.zst | zstd -d > /dev/null works | 07:07:26 |
| Sergei Zimmerman (xokdvium) | Huh, I was able to repro just now but no longer | 07:08:35 |
| c2fc2f left the room. | 07:09:27 |
 dramforever | wfm... cdn inconsistency? | 07:09:37 |
| c2fc2f joined the room. | 07:09:46 |
| Vladimír Čunát | If a builder needs a path which is missing in S3, it will build and upload it, if I understand the current implementation right. | 07:11:46 |
| Vladimír Čunát | That helps recovery. But we have some leaf builds left which show succeeded on Hydra but don't have a (usable) result in S3. | 07:12:52 |
 K900 | Not sure why, but the build logs no longer strip ANSI escapes | 07:18:14 |
| K900 | Might have something to do with how the new runner calls the builds? | 07:18:25 |
| Vladimír Čunát | Tracked in https://github.com/NixOS/hydra/issues/1802 | 07:20:52 |
| K900 | Honestly rendering on the frontend should be pretty doable with something like https://github.com/drudru/ansi_up | 07:24:07 |
| dramforever | speaking of, i just realized that the build logs doesn't seem to be uploaded to cache.nixos.org anymore. is this normal? | 07:24:08 |
| dramforever | * speaking of logs, i just realized that the build logs doesn't seem to be uploaded to cache.nixos.org anymore. is this normal? | 07:24:13 |
| K900 | (I think that one is what gitlab CI is using) | 07:24:16 |
 Mic92 | Looks like clients are racing to upload nars, which wouldn't be a problem for nix but than fastly caches different filesizes because of non-reproducible compression, which produces the error we see. | 07:26:17 |
| dramforever | for example, this is a build https://hydra.nixos.org/build/331688229/nixlog/160 for /nix/store/wvhfhvkgb4qwlhl5d32sjrr9bw13g02b-nixos-graphical-26.11pre1019198.459bc06669f8-x86_64-linux.iso.drv, with output /nix/store/9rgmsc5vxymhmj2jf8rmz3m52f49j038-nixos-graphical-26.11pre1019198.459bc06669f8-x86_64-linux.iso. The latter is on c.n.o, but https://cache.nixos.org/log/wvhfhvkgb4qwlhl5d32sjrr9bw13g02b-nixos-graphical-26.11pre1019198.459bc06669f8-x86_64-linux.iso.drv is 404 | 07:26:22 |
| Mic92 | To fix the existing errors, we can flush the fastly cache. | 07:27:13 |
| Vladimír Čunát | That sounds like a design problem? We need to lock each .drv hash in the queue-runner then? | 07:27:49 |
| Mic92 | and the hydra fix would be s3 upload + If-None-Match | 07:27:59 |
| Mic92 | so it's write once | 07:28:07 |
| Mic92 | The looser will try to download the nar as well in case the original uploader never returns | 07:28:27 |
| K900 | OK actually it's like five lines of code | 07:28:59 |
| K900 | Let me just do a PR | 07:29:01 |
| Vladimír Čunát | I still wouldn't be too happy if a builder built stuff against a local version of a dependency instead of waiting for the official one which ends up in S3. | 07:29:29 |
| Vladimír Čunát | * I still wouldn't be too happy if a builder built stuff against a local version of a dependency instead of waiting for the "official one" which ends up in S3. | 07:29:40 |
| Vladimír Čunát | Smells like rare weird effects if for stuff which isn't 100% binary reproducible. | 07:30:15 |
