| 19 Mar 2026 |
 Mic92 | aws, doesn't seem to use this interface | 12:28:30 |
 Sergei Zimmerman (xokdvium) | Well it effectively does the same thing. What aws probably does is retry on that 400 error | 12:29:30 |
 Arian | In reply to @joerg:thalheim.io
but aws does it's own pooling Only if you use the transfer API iirc which we didn't | 12:32:54 |
| Arian | For S3 you also need to retry on 503 for uploads
https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-design-patterns.html | 12:34:08 |
| Arian | They use 503 for rate limit :') | 12:34:30 |
| Mic92 | https://github.com/aws/aws-sdk-cpp/blob/9204e236faaa1ca6a0342dee7caf61c7cf5ad8bb/src/aws-cpp-sdk-core/source/client/CoreErrors.cpp#L90-L91 | 12:35:08 |
| Sergei Zimmerman (xokdvium) | Yup bernardo added that in https://github.com/NixOS/nix/pull/15449 | 12:35:11 |
| Arian | Sweet | 12:35:38 |
| Mic92 | where dos this handle 400? | 12:37:33 |
| Mic92 | * where does this handle 400? | 12:37:39 |
| Sergei Zimmerman (xokdvium) | I was talking about 503, 400 is not handled there. We'd need to look at the response xml | 12:38:17 |
| Arian | Line 113 | 12:38:56 |
| Arian | Decides if an error is retry able | 12:39:11 |
| Mic92 | True, but I think Sergei is right, we need to look at the xml | 12:41:00 |
| Mic92 | deploying this now | 12:44:59 |
| Mic92 | how did you came up with this number, so I can monitor? | 12:49:53 |
 hexa | journalctl -u hydra-queue-runner --since "2 days ago" --grep="unable to upload" | wc -l | 12:50:38 |
| Mic92 | okay, we don't have any upload error since: Thu Mar 19 01:01:45 PM UTC 2026 | 13:02:00 |
| Mic92 | Let's see | 13:02:05 |
| Mic92 | journalctl -u hydra-queue-runner --since "3 hour ago" --grep="unable to upload" is also empty | 13:02:13 |
| Mic92 | while true; do journalctl -u hydra-queue-runner --since "2026-03-19 13:01:45" --grep="unable to upload"; sleep 180; done | 13:06:19 |
| Mic92 | * while true; do journalctl -u hydra-queue-runner --since "2026-03-19 13:01:45" --grep="unable to upload"; sleep 180; done for monitoring | 13:06:29 |
| Mic92 | so far only getting a bunch of 500er:
Mar 19 13:11:55 mimas hydra-queue-runner[2800047]: warning: unable to upload 'https://nix-cache.s3.us-east-1.amazonaws.com/nar/0a8jn6fw74pdnpa10viam4wb4jmpszkfw19zrrm1h1n40zlpnlcx.nar.xz': HTTP error 503
Mar 19 13:11:55 mimas hydra-queue-runner[2800047]: warning: unable to upload 'https://nix-cache.s3.us-east-1.amazonaws.com/nar/0a8jn6fw74pdnpa10viam4wb4jmpszkfw19zrrm1h1n40zlpnlcx.nar.xz': HTTP error 503
| 13:16:20 |
| Mic92 | * so far only getting a bunch of one 500er:
Mar 19 13:11:55 mimas hydra-queue-runner[2800047]: warning: unable to upload 'https://nix-cache.s3.us-east-1.amazonaws.com/nar/0a8jn6fw74pdnpa10viam4wb4jmpszkfw19zrrm1h1n40zlpnlcx.nar.xz': HTTP error 503 | 13:16:40 |
| Mic92 | Still only one error. | 15:24:49 |
| Mic92 | Backport of the fix: https://github.com/NixOS/nix/pull/15525 | 15:58:59 |
| Mic92 | hexa (signing key rotation when): vcunat Jeremy Fleischman (jfly) dgrig https://meet.cccda.de/nix-osin-fra starts in 5min according to my calendar Arian already implied he can't join today. | 16:56:00 |
| hexa | huh, arian implied he could demo the mdm stuff earlier today | 16:57:43 |
| Arian | Just wrote in the infra team chat. I have a monthly team meeting that overlaps :( | 16:59:08 |
| hexa | ok | 16:59:17 |
