| 19 Mar 2026 |
 Vladimír Čunát | Hydra now shows lots of abortions from S3 uploads.
Aborted: [31;1merror:[0m … while uploading to S3 binary cache at '[35;1ms3://nix-cache[0m' [31;1merror:[0m unable to upload '[35;1mhttps://nix-cache.s3.us-east-1.amazonaws.com/nar/0fi6dj94id8fla40n3h98b9spmsbszl7i7k2bkpnjbqk9fcgyad8.nar.xz[0m': HTTP error [35;1m400[0m[35;1m[0m response body: [35;1m<?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>2SDA6N47EBP0Y9TY</RequestId><HostId>DzalNia2QC6pPMWG4tSso2PO0E+w+ZmqUrCtA+oI2GP1hMZV3IVW5WibfNELfak4rQVKuh6OrCw=</HostId></Error>[0m
| 08:08:00 |
 hexa | RequestTimeout
| 08:08:37 |
| hexa | hydra surely retries uplodas, right? | 08:08:50 |
| Vladimír Čunát | The Hydra jobs certainly remained failed/aborted until I restarted them manually. | 08:11:34 |
 Sergei Zimmerman (xokdvium) | The queue runner is at 2.34 now? | 08:59:09 |
| hexa | Hydra 0.1.20260316.a40d428 (using nix-2.34.1 and nix-eval-jobs-2.34.1)
| 09:03:48 |
| hexa | Mar 17 16:25:15 mimas hydra-queue-runner[4333]: warning: error: unable to upload 'https://s3.us-east-1.amazonaws.com/nix-cache/yak3aj118k761cgsnc79k0wpk4y5598c.ls': Timeout was reached (28) Operation timed out after 15942 milliseconds with 0 bytes received; retrying in 287 ms
they started here | 09:04:57 |
| hexa | and there have been ~17.8k of these since | 09:05:14 |
| hexa | that coincides with https://github.com/NixOS/infra/pull/978 | 09:06:09 |
| Vladimír Čunát | I've seen these abortions occasionally in the past few months. | 09:08:09 |
| Vladimír Čunát | * I've seen these S3 abortions occasionally in the past few months. | 09:08:19 |
| Sergei Zimmerman (xokdvium) | So 2.33 wasn’t experiencing this? The only meaningful change since then I think was adding tcp keepalive | 09:08:21 |
| Sergei Zimmerman (xokdvium) | And also switching to virtual-hosted–style endpoints. Might have made a difference there | 09:17:57 |
| Sergei Zimmerman (xokdvium) | Connection is most likely the issue here. As a quickfix I think this should help:
https://github.com/NixOS/infra/blob/22a38e7c5f61d7b04231c3dd84f5ca1a6fec52ef/build/hydra.nix#L88
Adding addressing-style=path should prevent connection reuse since it's http1-only.
| 09:27:19 |
| Sergei Zimmerman (xokdvium) | And s3 sends a connection close on those always. We'll see about making connection reuse much less aggressive and do a patch release of 2.34 | 09:28:06 |
| hexa | applying | 09:34:47 |
 Arian | S3 doesn't support http2 at all AFAIK? | 09:42:35 |
| Arian | Fwiw S3 explicitly documents that the way to scale it is to open multiple TCP connections. The A record returns like 50 different IP addresses and should connect to all of them | 09:43:05 |
| Sergei Zimmerman (xokdvium) | It does on the virtual-hosted buckets apparently | 09:43:13 |
| Vladimír Čunát | Sounds like horrible design. | 09:43:51 |
| Arian | That would be news to me. S3 has been very optimized for http 1.1 | 09:43:59 |
| Arian | Well it's the oldest Amazon service... | 09:44:03 |
| Sergei Zimmerman (xokdvium) | People did start screaming about multiple h1.1 connections:
https://github.com/NixOS/nix/issues/15208 | 09:48:42 |
| Vladimír Čunát | Those changes do sound related. | 09:49:56 |
| Arian |
arian@Arians-Mac-mini:~/ > curl https://sudachi.s3-ap-northeast-1.amazonaws.com --head
HTTP/1.1 200 OK
| 09:51:04 |
| Arian | I dont see HTTP2. And I would be extremely surprised if AWS ever will support HTTP2 on S3. | 09:52:28 |
| Arian | Yeh that seems related. We should definitely configure curl to reuse HTTP 1.1 connections. I thought it did that automatically? | 09:52:56 |
| Arian | This seems like an AI hallucination to me IMO: https://github.com/NixOS/nix/commit/695501815b0f8fb6e51ccf6598701fc459f9328f | 09:54:00 |
| Arian | This whole commit message is bogus. It’s not fixing the issue because S3 does not support HTTP2. you need to have a HTTP client that reuses connections according to the HTTP 1.1 spec | 09:54:57 |
| Vladimír Čunát |
And I would be extremely surprised if AWS ever will support HTTP2 on S3.
Perhaps skip directly to QUIC 😁
| 09:54:59 |
