| 19 Jun 2026 |
 dramforever | yeah but i need to sleep 🥲 | 14:56:29 |
 Mic92 | * Okay nixos-manual hydra product rebuilds correctly from the binary cache | 14:57:25 |
| Mic92 | hexa (signing key rotation when): I will now go ahead and first all builder and than hydra | 14:57:56 |
| dramforever | ("we" don't re-download paths that already have a narinfo, so if none of the store paths have made it into the closure of a new store-paths.xz, then all we need to do after is restart and go back to normal polling) | 14:59:08 |
| Mic92 | That's the only one I knew that was new: /nix/store/lq8laawx5ii4jrs3rcvyjya3gvk5d6lq-ruby3.4-serverengine-2.4.0
but I think I deleted it from the binary cache before any channel bump | 15:02:25 |
| dramforever | yes, that and /nix/store/9wzl5nkynf6727nabm2kjq523a57qyin-gst-libav-1.26.11 | 15:03:18 |
| dramforever | afaict. neither of which has been pulled to TUNA/ISCAS yet | 15:03:41 |
| dramforever | as i would expect if the channels have not bumped to include them yet | 15:04:42 |
| dramforever | so, good sign | 15:04:49 |
|  Shahar "Dawn" Or joined the room. | 15:05:33 |
| Mic92 | If someone has some spare cycles going over store-paths.xz would be a different way to vet the cache | 15:06:02 |
| dramforever | 🥲 we would have been doing that if i hadn't got lazy 6.5 years ago and skipped on checking FileHash | 15:08:09 |
| Mic92 | Maybe also should do a GC on all builders before taking them back online? | 15:26:09 |
 hexa | looks like we're back up | 15:56:03 |
| Shahar "Dawn" Or | Thank you for bringing it back up so quickly 🙏 | 16:05:14 |
| Mic92 | I think I was able to observe this now. Apparently if a multi-part complete request returns an error, one has to check if the object was created successfully. Our retry code was just retrying on already invalidated part ids. | 16:29:27 |
| Mic92 | New version is deployed | 16:29:39 |
| Mic92 | Looks like hydra's disk no longer get trashed with nars... good so far. However NAR streaming is still quiet heavy and blocks the queue-runner async code a bit, so I should get this out of the event loop. My first attempt will be switching to ls files since the vast majority of uncached nars won't have any hydra build productions. | 17:03:03 |
| Mic92 | * Looks like hydra's disk no longer get trashed with nars... good so far. However NAR streaming is still quiet heavy and blocks the queue-runner async code a bit, so I should get this out of the event loop. My first attempt will be switching to ls files since the vast majority of uncached nars won't have any hydra build products. | 17:03:20 |
| Mic92 | * Looks like hydra's disk no longer get trashed with nars... good so far. However NAR streaming is still quiet heavy and blocks the queue-runner async code a bit, so I should get this out of the event loop. My first attempt will be switching to ls files since the vast majority of uncached nars won't have any hydra build products -> than no decompression is required. | 17:04:22 |
| Mic92 | However CPU usage just looks okay, so I will not deploy for now and let it instead get through the backlog. Instead I am going to test it a bit on staging hydra. | 17:28:30 |
| Mic92 | Okay. Signing out for today. I will have some time tomorrow for smaller fixes but not on Sunday. | 17:33:32 |
| hexa | Thanks! | 17:34:12 |
| hexa | 
Download | 20:33:49 |
| hexa | I think we're not quite there yet with uploads to s3 🤔 | 20:33:59 |
 John Ericson | long term I hope to make us not use the queue runner store entirely | 21:22:02 |
| John Ericson | everything should be binary cache or database | 21:22:12 |
| John Ericson | and evaluations should be distributed to builders just like regular build | 21:22:21 |
|  morwdan joined the room. | 23:16:29 |
| 20 Jun 2026 |
| Mic92 | @hexa:lossy.network: when see a lot of receiving uploads, check also the load on the machine | 04:19:38 |
