| 19 Jun 2026 |
 hexa (signing key rotation when) | Redacted or Malformed Event | 14:43:55 |
 Mic92 | Okay, I guess I have to drop the DNS change in this case | 14:44:10 |
| hexa (signing key rotation when) | you can bypass the merge queue | 14:44:30 |
| hexa (signing key rotation when) | whenever it runs again it will retry applying it | 14:44:34 |
| hexa (signing key rotation when) | the merge queue will block dns changes until gandi gets that fixed | 14:44:52 |
| hexa (signing key rotation when) | 
Download | 14:45:38 |
| hexa (signing key rotation when) | no outage for iocaine customers 😉 | 14:45:46 |
| hexa (signing key rotation when) | and tagging bots for the maze is working | 14:46:06 |
| Mic92 | So my plan is to quickly test the build-support code in staging and than do a deployment of hydra + builder again | 14:48:09 |
| hexa (signing key rotation when) | ok | 14:48:27 |
| Mic92 | I had now for a couple of hour staging running and restarted builders to see if they leave builds behind in the db | 14:48:28 |
 dramforever | i checked the "preliminary list" on the github issue and it looks like neither TUNA nor ISCAS has grabbed any of the wrong store paths. it seems that none of the new? store paths have made it to a channel bump? | 14:55:12 |
| Mic92 | Okay nixos-manual rebuilds correctly from the binary cache | 14:55:13 |
| dramforever | knocking on wood but if so that would be a relief | 14:55:36 |
| Mic92 | Okay, I didn't finished the parquet file fetch yet. | 14:56:08 |
| dramforever | yeah but i need to sleep 🥲 | 14:56:29 |
| Mic92 | * Okay nixos-manual hydra product rebuilds correctly from the binary cache | 14:57:25 |
| Mic92 | hexa (signing key rotation when): I will now go ahead and first all builder and than hydra | 14:57:56 |
| dramforever | ("we" don't re-download paths that already have a narinfo, so if none of the store paths have made it into the closure of a new store-paths.xz, then all we need to do after is restart and go back to normal polling) | 14:59:08 |
| Mic92 | That's the only one I knew that was new: /nix/store/lq8laawx5ii4jrs3rcvyjya3gvk5d6lq-ruby3.4-serverengine-2.4.0
but I think I deleted it from the binary cache before any channel bump | 15:02:25 |
| dramforever | yes, that and /nix/store/9wzl5nkynf6727nabm2kjq523a57qyin-gst-libav-1.26.11 | 15:03:18 |
| dramforever | afaict. neither of which has been pulled to TUNA/ISCAS yet | 15:03:41 |
| dramforever | as i would expect if the channels have not bumped to include them yet | 15:04:42 |
| dramforever | so, good sign | 15:04:49 |
|  Shahar "Dawn" Or joined the room. | 15:05:33 |
| Mic92 | If someone has some spare cycles going over store-paths.xz would be a different way to vet the cache | 15:06:02 |
| dramforever | 🥲 we would have been doing that if i hadn't got lazy 6.5 years ago and skipped on checking FileHash | 15:08:09 |
| Mic92 | Maybe also should do a GC on all builders before taking them back online? | 15:26:09 |
| hexa (signing key rotation when) | looks like we're back up | 15:56:03 |
| Shahar "Dawn" Or | Thank you for bringing it back up so quickly 🙏 | 16:05:14 |
