| 3 Jul 2026 |
 Alyssa Ross | are any of the broken things mass rebuilds? | 13:30:05 |
 emily | so just whack-a-moling those probably helps a lot | 13:30:19 |
 K900 | Well some of them are | 13:30:24 |
| emily | I do worry about mass breakage in leaves, but it seems to mostly be "big" packages that fail | 13:30:29 |
| Alyssa Ross | presumably most of them don't need to be fixed in staging | 13:30:31 |
| emily | my guess is that it happens when you link too many files or something | 13:30:41 |
| K900 | Yeah but then we have to revert all of those hacks on staging again | 13:30:45 |
| K900 | And that's human time | 13:30:54 |
| K900 | Which is more valuable than Hydra time | 13:30:59 |
| Alyssa Ross | (I'm long of the opinion that we shouldn't delay staging to fix leafs. It just perpetuates the concentration of work onto a few overstretched people.) | 13:31:24 |
| emily | well, we're at like 4, 5 PRs I already need to revert? I can revert a few more. if it ended up being tons of uncoordinated hacks in leaf packages then I'd be sad | 13:31:26 |
| emily | to be clear, it's been tons and tons of critical packages that break, e.g. GLib GTK Qt SDL FFmpeg Swift matplotlib … | 13:32:01 |
| emily | working around each one isn't too bad though | 13:32:19 |
| Alyssa Ross | I had misread this | 13:32:31 |
| emily | I just hope that it's primarily something that breaks those big packages and not everything downstream too | 13:32:33 |
| Alyssa Ross | Is that a testable hypothesis? Build loads of glib dependents overnight and see if they're fine? | 13:37:54 |
| Alyssa Ross | or even check dependents on Hydra of big stuff that's already been fixed? | 13:38:13 |
| emily | well, the test results are so far is that we are frequently fixing stuff and then seeing big stuff close beneath it fail | 13:40:31 |
| emily | I don't think it's worth rerolling though | 13:40:37 |
| emily | it just sucks to triage | 13:40:41 |
 Vladimír Čunát | They hide 15k builds. No idea how many individual breakages lurk underneath. | 15:01:24 |
| emily | right | 15:01:37 |
| Vladimír Čunát | Hydra doesn't anything to do for darwin currently. | 15:03:05 |
| Vladimír Čunát | I could start a staging-next-26.05, though. There's still more darwin in there. | 15:03:37 |
| Vladimír Čunát | * I could start a parallel staging-next-26.05, though. There's still more darwin in there. | 15:03:42 |
| Vladimír Čunát | (and much less human work surely) | 15:03:57 |
| Vladimír Čunát | Or we could roll the darwin rebuild (or possibly whole staging). | 15:04:34 |
| emily | I can get fixes up for the big blockers | 15:05:07 |
| emily | in like 30-60 minutes | 15:05:20 |
| Vladimír Čunát | Or that. | 15:05:24 |
