| 23 Jun 2026 |
 Mic92 | https://github.com/NixOS/nix-eval-jobs/pull/422 | 20:43:50 |
 Austin Horstman | Wasn't sure where to ask, but I'll reach out in here on a question I have related to pkgsCross and that releae-cross hydra set. We were adding wasm support for tree-sitter and I was recommended to add the set to the release-cross so that hydra builds and caches the wasm grammars. It's about ~350 per platform. Is that an issue for increasing hydra build counts for cross? https://github.com/NixOS/nixpkgs/pull/534687 | 21:33:46 |
| Austin Horstman | * Wasn't sure where to ask, but I'll reach out in here on a question I have related to pkgsCross and that releae-cross hydra set. We were adding wasm support for tree-sitter and I was recommended to add the set to the release-cross so that hydra builds and caches the wasm grammars. It's about ~300 per platform. Is that an issue for increasing hydra build counts for cross? https://github.com/NixOS/nixpkgs/pull/534687 | 21:34:16 |
| 24 Jun 2026 |
 dramforever | "per platform" isn't a big deal if the number of platforms is 1 | 01:36:31 |
| dramforever | the release-cross file is organized by platforms. you want to add it to wasiCommon https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/release-cross.nix | 01:37:33 |
| Austin Horstman | ah, thanks. i'll tweak it. i added it to wasi32 at the bottom instead. | 01:44:39 |
 Vladimír Čunát |
error: failed to read compressed data (Zstd decompression failed: Data corruption detected) error: path /nix/store/xbpvckqw1v2bg2p1r6jl6s3viny9rjiv-clang-src-22.1.8 is required, but there is no substituter that can build it
We have these failures in several packages, blocked thousands of jobs.
| 05:50:38 |
| Vladimír Čunát | *
error: failed to read compressed data (Zstd decompression failed: Data corruption detected) error: path /nix/store/xbpvckqw1v2bg2p1r6jl6s3viny9rjiv-clang-src-22.1.8 is required, but there is no substituter that can build it
We have these failures in several packages, blocked at least a thousand jobs.
| 05:50:58 |
| Vladimír Čunát | When I fetch the path now locally (nix build /nix/store...), it succeeds. | 05:52:43 |
| Vladimír Čunát | But I can reproduce this one
$ nix build /nix/store/pbgwd0c61p5p0c1fmklx30wqk7q33nj5-rusty-v8-149.3.0-vendor
warning: error: unable to download 'https://cache.nixos.org/nar/1imr6zrqbb8l9l933yhsxsxl7j0ygs4prrrn50aiicr3hxp23n0w.nar.zst': HTTP error 200 (curl error: Stream error in the HTTP/2 framing layer); retrying from offset 8388608 in 253 ms
error: failed to read compressed data (Zstd decompression failed: Data corruption detected)
error: path '/nix/store/pbgwd0c61p5p0c1fmklx30wqk7q33nj5-rusty-v8-149.3.0-vendor' is required, but there is no substituter that can build it
| 05:53:57 |
|  c2fc2f joined the room. | 06:48:51 |
 Sergei Zimmerman (xokdvium) | Huh, doing curl https://cache.nixos.org/nar/1imr6zrqbb8l9l933yhsxsxl7j0ygs4prrrn50aiicr3hxp23n0w.nar.zst | zstd -d > /dev/null works | 07:07:26 |
| Sergei Zimmerman (xokdvium) | Huh, I was able to repro just now but no longer | 07:08:35 |
| c2fc2f left the room. | 07:09:27 |
| dramforever | wfm... cdn inconsistency? | 07:09:37 |
| c2fc2f joined the room. | 07:09:46 |
| Vladimír Čunát | If a builder needs a path which is missing in S3, it will build and upload it, if I understand the current implementation right. | 07:11:46 |
| Vladimír Čunát | That helps recovery. But we have some leaf builds left which show succeeded on Hydra but don't have a (usable) result in S3. | 07:12:52 |
 K900 | Not sure why, but the build logs no longer strip ANSI escapes | 07:18:14 |
| K900 | Might have something to do with how the new runner calls the builds? | 07:18:25 |
| Vladimír Čunát | Tracked in https://github.com/NixOS/hydra/issues/1802 | 07:20:52 |
| K900 | Honestly rendering on the frontend should be pretty doable with something like https://github.com/drudru/ansi_up | 07:24:07 |
| dramforever | speaking of, i just realized that the build logs doesn't seem to be uploaded to cache.nixos.org anymore. is this normal? | 07:24:08 |
| dramforever | * speaking of logs, i just realized that the build logs doesn't seem to be uploaded to cache.nixos.org anymore. is this normal? | 07:24:13 |
| K900 | (I think that one is what gitlab CI is using) | 07:24:16 |
| Mic92 | Looks like clients are racing to upload nars, which wouldn't be a problem for nix but than fastly caches different filesizes because of non-reproducible compression, which produces the error we see. | 07:26:17 |
| dramforever | for example, this is a build https://hydra.nixos.org/build/331688229/nixlog/160 for /nix/store/wvhfhvkgb4qwlhl5d32sjrr9bw13g02b-nixos-graphical-26.11pre1019198.459bc06669f8-x86_64-linux.iso.drv, with output /nix/store/9rgmsc5vxymhmj2jf8rmz3m52f49j038-nixos-graphical-26.11pre1019198.459bc06669f8-x86_64-linux.iso. The latter is on c.n.o, but https://cache.nixos.org/log/wvhfhvkgb4qwlhl5d32sjrr9bw13g02b-nixos-graphical-26.11pre1019198.459bc06669f8-x86_64-linux.iso.drv is 404 | 07:26:22 |
| Mic92 | To fix the existing errors, we can flush the fastly cache. | 07:27:13 |
| Vladimír Čunát | That sounds like a design problem? We need to lock each .drv hash in the queue-runner then? | 07:27:49 |
| Mic92 | and the hydra fix would be s3 upload + If-None-Match | 07:27:59 |
