| 19 Jun 2026 |
 dramforever | as i would expect if the channels have not bumped to include them yet | 15:04:42 |
| dramforever | so, good sign | 15:04:49 |
|  Shahar "Dawn" Or joined the room. | 15:05:33 |
 Mic92 | If someone has some spare cycles going over store-paths.xz would be a different way to vet the cache | 15:06:02 |
| dramforever | 🥲 we would have been doing that if i hadn't got lazy 6.5 years ago and skipped on checking FileHash | 15:08:09 |
| Mic92 | Maybe also should do a GC on all builders before taking them back online? | 15:26:09 |
 hexa | looks like we're back up | 15:56:03 |
| Shahar "Dawn" Or | Thank you for bringing it back up so quickly 🙏 | 16:05:14 |
| Mic92 | I think I was able to observe this now. Apparently if a multi-part complete request returns an error, one has to check if the object was created successfully. Our retry code was just retrying on already invalidated part ids. | 16:29:27 |
| Mic92 | New version is deployed | 16:29:39 |
| Mic92 | Looks like hydra's disk no longer get trashed with nars... good so far. However NAR streaming is still quiet heavy and blocks the queue-runner async code a bit, so I should get this out of the event loop. My first attempt will be switching to ls files since the vast majority of uncached nars won't have any hydra build productions. | 17:03:03 |
| Mic92 | * Looks like hydra's disk no longer get trashed with nars... good so far. However NAR streaming is still quiet heavy and blocks the queue-runner async code a bit, so I should get this out of the event loop. My first attempt will be switching to ls files since the vast majority of uncached nars won't have any hydra build products. | 17:03:20 |
| Mic92 | * Looks like hydra's disk no longer get trashed with nars... good so far. However NAR streaming is still quiet heavy and blocks the queue-runner async code a bit, so I should get this out of the event loop. My first attempt will be switching to ls files since the vast majority of uncached nars won't have any hydra build products -> than no decompression is required. | 17:04:22 |
| Mic92 | However CPU usage just looks okay, so I will not deploy for now and let it instead get through the backlog. Instead I am going to test it a bit on staging hydra. | 17:28:30 |
| Mic92 | Okay. Signing out for today. I will have some time tomorrow for smaller fixes but not on Sunday. | 17:33:32 |
| hexa | Thanks! | 17:34:12 |
| hexa | 
Download | 20:33:49 |
| hexa | I think we're not quite there yet with uploads to s3 🤔 | 20:33:59 |
 John Ericson | long term I hope to make us not use the queue runner store entirely | 21:22:02 |
| John Ericson | everything should be binary cache or database | 21:22:12 |
| John Ericson | and evaluations should be distributed to builders just like regular build | 21:22:21 |
|  morwdan joined the room. | 23:16:29 |
| 20 Jun 2026 |
| Mic92 | @hexa:lossy.network: when see a lot of receiving uploads, check also the load on the machine | 04:19:38 |
| Mic92 | Because if all CPUs are working, I don't think we can make compression go much faster | 04:20:17 |
| Mic92 | Yesterday I reduced the number of concurrent uploads a bit again because it was causing issues on some smaller machines. I want to optimise a bit more for stability first before going for maximum throughput | 04:21:53 |
| Mic92 | * @hexa:lossy.network: when you see a lot of receiving uploads, check also the load on the machine | 04:22:33 |
| Mic92 | The current upload code for S3 is also not optimal. Usually one should auto scale connections based on what the S3 store respond in terms of error codes | 04:25:04 |
| Mic92 | I would like to get log and open telemetry collection to work. Than we get more insights | 04:26:21 |
| Mic92 | * The current upload code for S3 is also not optimal. Usually one should auto scale connections based on what the S3 service respond in terms of error codes | 04:26:42 |
 Vladimír Čunát | I wondered why big-parallel builds are in so much trouble since the switch, and apparently we build them with -j1 | 05:56:57 |
