| 19 Jun 2026 |
 hexa | ok | 14:48:27 |
 Mic92 | I had now for a couple of hour staging running and restarted builders to see if they leave builds behind in the db | 14:48:28 |
 dramforever | i checked the "preliminary list" on the github issue and it looks like neither TUNA nor ISCAS has grabbed any of the wrong store paths. it seems that none of the new? store paths have made it to a channel bump? | 14:55:12 |
| Mic92 | Okay nixos-manual rebuilds correctly from the binary cache | 14:55:13 |
| dramforever | knocking on wood but if so that would be a relief | 14:55:36 |
| Mic92 | Okay, I didn't finished the parquet file fetch yet. | 14:56:08 |
| dramforever | yeah but i need to sleep 🥲 | 14:56:29 |
| Mic92 | * Okay nixos-manual hydra product rebuilds correctly from the binary cache | 14:57:25 |
| Mic92 | hexa (signing key rotation when): I will now go ahead and first all builder and than hydra | 14:57:56 |
| dramforever | ("we" don't re-download paths that already have a narinfo, so if none of the store paths have made it into the closure of a new store-paths.xz, then all we need to do after is restart and go back to normal polling) | 14:59:08 |
| Mic92 | That's the only one I knew that was new: /nix/store/lq8laawx5ii4jrs3rcvyjya3gvk5d6lq-ruby3.4-serverengine-2.4.0
but I think I deleted it from the binary cache before any channel bump | 15:02:25 |
| dramforever | yes, that and /nix/store/9wzl5nkynf6727nabm2kjq523a57qyin-gst-libav-1.26.11 | 15:03:18 |
| dramforever | afaict. neither of which has been pulled to TUNA/ISCAS yet | 15:03:41 |
| dramforever | as i would expect if the channels have not bumped to include them yet | 15:04:42 |
| dramforever | so, good sign | 15:04:49 |
|  Shahar "Dawn" Or joined the room. | 15:05:33 |
| Mic92 | If someone has some spare cycles going over store-paths.xz would be a different way to vet the cache | 15:06:02 |
| dramforever | 🥲 we would have been doing that if i hadn't got lazy 6.5 years ago and skipped on checking FileHash | 15:08:09 |
| Mic92 | Maybe also should do a GC on all builders before taking them back online? | 15:26:09 |
| hexa | looks like we're back up | 15:56:03 |
| Shahar "Dawn" Or | Thank you for bringing it back up so quickly 🙏 | 16:05:14 |
| Mic92 | I think I was able to observe this now. Apparently if a multi-part complete request returns an error, one has to check if the object was created successfully. Our retry code was just retrying on already invalidated part ids. | 16:29:27 |
| Mic92 | New version is deployed | 16:29:39 |
| Mic92 | Looks like hydra's disk no longer get trashed with nars... good so far. However NAR streaming is still quiet heavy and blocks the queue-runner async code a bit, so I should get this out of the event loop. My first attempt will be switching to ls files since the vast majority of uncached nars won't have any hydra build productions. | 17:03:03 |
| Mic92 | * Looks like hydra's disk no longer get trashed with nars... good so far. However NAR streaming is still quiet heavy and blocks the queue-runner async code a bit, so I should get this out of the event loop. My first attempt will be switching to ls files since the vast majority of uncached nars won't have any hydra build products. | 17:03:20 |
| Mic92 | * Looks like hydra's disk no longer get trashed with nars... good so far. However NAR streaming is still quiet heavy and blocks the queue-runner async code a bit, so I should get this out of the event loop. My first attempt will be switching to ls files since the vast majority of uncached nars won't have any hydra build products -> than no decompression is required. | 17:04:22 |
| Mic92 | However CPU usage just looks okay, so I will not deploy for now and let it instead get through the backlog. Instead I am going to test it a bit on staging hydra. | 17:28:30 |
| Mic92 | Okay. Signing out for today. I will have some time tomorrow for smaller fixes but not on Sunday. | 17:33:32 |
| hexa | Thanks! | 17:34:12 |
| hexa | 
Download | 20:33:49 |
