| 18 Jun 2026 |
 Mic92 | In reply to @vcunat:matrix.org
The situation doesn't seem sustainable to me for more than a couple days. Even -small channels get blocked because of the abortions and retrying doesn't seem to help, e.g.:
https://hydra.nixos.org/build/331658358 Could please make issue reports and ping me here on them? Scraping from matrix is a bit of messy situation | 07:31:43 |
| Mic92 | Especially now that we have old and new reports in this thread | 07:33:52 |
| Mic92 | @r-burns:matrix.org: same for you | 07:35:08 |
| Mic92 | I will do another triage now to check if I can identify more sources of aborts just now. But but the next proper fixing session has to wait for the evening. I would do it in a staging environment, but my Aws servers didn't show a lot of the symptoms we are seeing now | 07:38:34 |
 Vladimír Čunát | Posted as
https://github.com/NixOS/hydra/issues/1804
https://github.com/NixOS/hydra/issues/1803 | 07:39:29 |
| Vladimír Čunát | Feel free to edit, add info, etc. | 07:39:41 |
| Vladimír Čunát | To recover we simply remove this NAR+narinfo from S3, right? | 07:43:12 |
| Vladimír Čunát | As it's a fixed-output derivation, it should be cheap to "build" locally, and I expect Hydra will fill it up at some point automatically anyway if it will be missing in S3. | 07:43:47 |
| Mic92 | Yes. | 07:44:02 |
| Vladimír Čunát | * As it's a fixed-output derivation, it should be cheap to "build" locally, and I expect Hydra will fill it up at some point automatically anyway if it will be missing in S3. | 07:44:09 |
| Mic92 | I have a look at the logs first | 07:44:16 |
| Mic92 | and check if we I can root cause this | 07:44:20 |
| Vladimír Čunát | Sure. Though removing it from S3 hopefully wouldn't damage these traces. | 07:44:56 |
| Vladimír Čunát | It also seems likely that this wasn't the only damaged NAR. | 07:45:15 |
|  buterfloro joined the room. | 07:45:24 |
| Vladimír Čunát | (In my experience, most failures do not get reported.) | 07:45:44 |
| buterfloro | I've got full logs that I can provide on request, unlike the truncated ones in the Discourse discussion | 07:46:55 |
| buterfloro | * I've got full logs on my end that I can provide on request, unlike the truncated ones in the Discourse discussion | 07:47:14 |
 K900 | It's not really going to help much, we know the wrong thing is uploaded | 07:48:35 |
| K900 | Figuring out why would require looking at infra, not at your system | 07:48:47 |
| Vladimír Čunát | Also the Unsupported system type:
https://github.com/NixOS/hydra/issues/1805 | 07:53:22 |
| Mic92 | thx | 08:03:10 |
| Mic92 | https://github.com/NixOS/hydra/issues/1805#issuecomment-4739539396 This might explain a few of the aborts | 08:10:05 |
|  vmfunc joined the room. | 08:17:29 |
 Sergei Zimmerman (xokdvium) | https://matrix.to/#/!CTCrFzsBPYmDLmrja4%3A0upti.me/%24olty6gvo6UpfhC5zyAr0LPIlO3U6CffpDZGhIAS89Ho
Nixpkgs isn’t faring too well too | 08:18:56 |
| vmfunc | hiya, ci looks broken repo-wide right now on nixpkgs. the ci/pinned.json bootstrap can't realise its tooling, hash mismatch on nghttp3-1.15.0 from cache.nixos.org (specified 157s6q…, got 07svi8…), no substituter, which takes out lint / parse, lint / nixpkgs-vet and eval / x86_64-darwin. it's been hitting PRs at random depending on how warm the runner's cache is. seeing it on #532911, #532912, and my #532910. known? pretty sure this is linked to cache | 08:18:59 |
| vmfunc changed their profile picture. | 08:19:39 |
| Sergei Zimmerman (xokdvium) | In reply to @vmfunc:mozilla.org
hiya, ci looks broken repo-wide right now on nixpkgs. the ci/pinned.json bootstrap can't realise its tooling, hash mismatch on nghttp3-1.15.0 from cache.nixos.org (specified 157s6q…, got 07svi8…), no substituter, which takes out lint / parse, lint / nixpkgs-vet and eval / x86_64-darwin. it's been hitting PRs at random depending on how warm the runner's cache is. seeing it on #532911, #532912, and my #532910. known? pretty sure this is linked to cache https://github.com/NixOS/hydra/issues/1804 | 08:19:39 |
| vmfunc | i see | 08:20:27 |
| Sergei Zimmerman (xokdvium) | Whats really scary is that the nixpkgs didn’t get bumped in CI? | 08:23:12 |
