| 19 Jun 2026 |
 dramforever | yes, that and /nix/store/9wzl5nkynf6727nabm2kjq523a57qyin-gst-libav-1.26.11 | 15:03:18 |
| dramforever | afaict. neither of which has been pulled to TUNA/ISCAS yet | 15:03:41 |
| dramforever | as i would expect if the channels have not bumped to include them yet | 15:04:42 |
| dramforever | so, good sign | 15:04:49 |
|  Shahar "Dawn" Or joined the room. | 15:05:33 |
 Mic92 | If someone has some spare cycles going over store-paths.xz would be a different way to vet the cache | 15:06:02 |
| dramforever | 🥲 we would have been doing that if i hadn't got lazy 6.5 years ago and skipped on checking FileHash | 15:08:09 |
| Mic92 | Maybe also should do a GC on all builders before taking them back online? | 15:26:09 |
 hexa (signing key rotation when) | looks like we're back up | 15:56:03 |
| Shahar "Dawn" Or | Thank you for bringing it back up so quickly 🙏 | 16:05:14 |
| Mic92 | I think I was able to observe this now. Apparently if a multi-part complete request returns an error, one has to check if the object was created successfully. Our retry code was just retrying on already invalidated part ids. | 16:29:27 |
| Mic92 | New version is deployed | 16:29:39 |
| Mic92 | Looks like hydra's disk no longer get trashed with nars... good so far. However NAR streaming is still quiet heavy and blocks the queue-runner async code a bit, so I should get this out of the event loop. My first attempt will be switching to ls files since the vast majority of uncached nars won't have any hydra build productions. | 17:03:03 |
| Mic92 | * Looks like hydra's disk no longer get trashed with nars... good so far. However NAR streaming is still quiet heavy and blocks the queue-runner async code a bit, so I should get this out of the event loop. My first attempt will be switching to ls files since the vast majority of uncached nars won't have any hydra build products. | 17:03:20 |
| Mic92 | * Looks like hydra's disk no longer get trashed with nars... good so far. However NAR streaming is still quiet heavy and blocks the queue-runner async code a bit, so I should get this out of the event loop. My first attempt will be switching to ls files since the vast majority of uncached nars won't have any hydra build products -> than no decompression is required. | 17:04:22 |
| Mic92 | However CPU usage just looks okay, so I will not deploy for now and let it instead get through the backlog. Instead I am going to test it a bit on staging hydra. | 17:28:30 |
