| 9 Oct 2025 |
 Arian | (first primary database overload issues. Then failed network equipment) | 21:47:06 |
| 10 Oct 2025 |
 uep | 
Download image.png | 03:02:08 |
| uep | Hydra is clearly having Some Kind Of A Time.. #hugops to whoever's trying to make it work | 03:02:39 |
| uep | those builds seem to restart or roll back and get forgotten a lot.. | 03:03:23 |
| uep | and the most recent eval seems to have gone backwards with the others too. | 03:03:49 |
| uep | (they kept going back to "1" pending job, until this latest much bigger jump) | 03:04:25 |
 Vladimír Čunát | uep: I don't understand what you mean. Something seems wrong in particular? | 05:59:46 |
| uep | those grey numbers for unfinished jobs.. | 06:00:25 |
| uep | they were finished | 06:00:33 |
| Vladimír Čunát | So this is trunk-combined showing and someone did restart-all-failed retrospectively. | 06:00:34 |
| uep | yeah, i assume a lot of them are the same job | 06:01:00 |
| Vladimír Čunát | nixos-unstable is c9b6fb798541 for several hours at least. | 06:01:05 |
| uep | i know.. i've been running c9b6 since yesterday | 06:02:09 |
| Vladimír Čunát | The channel scripts have extra code to avoid the possibility of channels going backwards - exactly for similar situations. | 06:02:50 |
| uep | so, i assumed hydra had been restarted and forgotten some of those jobs since clearly they had already run | 06:03:19 |
| uep | but explicit restarts makes sense too | 06:03:46 |
| Vladimír Čunát | This doesn't happen automatically. | 06:04:06 |
| uep | * but explicit job restarts makes sense too | 06:04:10 |
| Vladimír Čunát | But I don't know who did it. No audits for that. | 06:04:16 |
| uep | yeah, to be clear, i wasn't worried about the channel going backwards, what i meant about going backwards was just the build count progress. | 06:07:54 |
| uep | but if that's because (say) there are flaky python tests again and just rerunning lets more things land in cache, ok | 06:08:44 |
| Vladimír Čunát | Hydra doesn't allow you to restart succeeded builds. | 06:09:09 |
| Vladimír Čunát | So this is a relatively common situation for us. I was just trying to find out if there's something more anomalous, but it doesn't sound so. | 06:11:52 |
| uep | yeah, i thought it was hydra being flaky rather than deliberately retrying failed, possibly flaky, builds for evals that were already complete.. because it's been like this all week. | 06:13:31 |
| uep | never mind, thanks for the clarification | 06:13:48 |
| 11 Oct 2025 |
 hexa | updating the m1 macs to 15.7.1 | 15:03:45 |
| hexa | done | 15:19:12 |
| 13 Oct 2025 |
 teutat3s | Could someone restart the electron 38 hydra job in trunk-combined? It looks like various jobs failed, this one ran out of disk space
https://hydra.nixos.org/build/310015857/nixlog/1 | 15:59:16 |
| teutat3s | It was also reported here: https://github.com/NixOS/nixpkgs/issues/451482 | 16:00:16 |
|  vivekanandan_ks joined the room. | 18:57:46 |
