| 19 Mar 2026 |
 hexa | Mar 17 16:25:15 mimas hydra-queue-runner[4333]: warning: error: unable to upload 'https://s3.us-east-1.amazonaws.com/nix-cache/yak3aj118k761cgsnc79k0wpk4y5598c.ls': Timeout was reached (28) Operation timed out after 15942 milliseconds with 0 bytes received; retrying in 287 ms
they started here | 09:04:57 |
| hexa | and there have been ~17.8k of these since | 09:05:14 |
| hexa | that coincides with https://github.com/NixOS/infra/pull/978 | 09:06:09 |
 Vladimír Čunát | I've seen these abortions occasionally in the past few months. | 09:08:09 |
| Vladimír Čunát | * I've seen these S3 abortions occasionally in the past few months. | 09:08:19 |
 Sergei Zimmerman (xokdvium) | So 2.33 wasn’t experiencing this? The only meaningful change since then I think was adding tcp keepalive | 09:08:21 |
| Sergei Zimmerman (xokdvium) | And also switching to virtual-hosted–style endpoints. Might have made a difference there | 09:17:57 |
| Sergei Zimmerman (xokdvium) | Connection is most likely the issue here. As a quickfix I think this should help:
https://github.com/NixOS/infra/blob/22a38e7c5f61d7b04231c3dd84f5ca1a6fec52ef/build/hydra.nix#L88
Adding addressing-style=path should prevent connection reuse since it's http1-only.
| 09:27:19 |
| Sergei Zimmerman (xokdvium) | And s3 sends a connection close on those always. We'll see about making connection reuse much less aggressive and do a patch release of 2.34 | 09:28:06 |
| hexa | applying | 09:34:47 |
 Arian | S3 doesn't support http2 at all AFAIK? | 09:42:35 |
| Arian | Fwiw S3 explicitly documents that the way to scale it is to open multiple TCP connections. The A record returns like 50 different IP addresses and should connect to all of them | 09:43:05 |
| Sergei Zimmerman (xokdvium) | It does on the virtual-hosted buckets apparently | 09:43:13 |
| Vladimír Čunát | Sounds like horrible design. | 09:43:51 |
| Arian | That would be news to me. S3 has been very optimized for http 1.1 | 09:43:59 |
| Arian | Well it's the oldest Amazon service... | 09:44:03 |
| Sergei Zimmerman (xokdvium) | People did start screaming about multiple h1.1 connections:
https://github.com/NixOS/nix/issues/15208 | 09:48:42 |
| Vladimír Čunát | Those changes do sound related. | 09:49:56 |
| Arian |
arian@Arians-Mac-mini:~/ > curl https://sudachi.s3-ap-northeast-1.amazonaws.com --head
HTTP/1.1 200 OK
| 09:51:04 |
| Arian | I dont see HTTP2. And I would be extremely surprised if AWS ever will support HTTP2 on S3. | 09:52:28 |
| Arian | Yeh that seems related. We should definitely configure curl to reuse HTTP 1.1 connections. I thought it did that automatically? | 09:52:56 |
| Arian | This seems like an AI hallucination to me IMO: https://github.com/NixOS/nix/commit/695501815b0f8fb6e51ccf6598701fc459f9328f | 09:54:00 |
| Arian | This whole commit message is bogus. It’s not fixing the issue because S3 does not support HTTP2. you need to have a HTTP client that reuses connections according to the HTTP 1.1 spec | 09:54:57 |
| Vladimír Čunát |
And I would be extremely surprised if AWS ever will support HTTP2 on S3.
Perhaps skip directly to QUIC 😁
| 09:54:59 |
| hexa | Redacted or Malformed Event | 09:55:22 |
| hexa | This seems like an AI hallucination to me IMO
That would be wild | 09:55:52 |
| hexa | This seems like an AI hallucination to me IMO
That would be wild | 09:55:55 |
| Arian | But the TCP-keepalive change does sound like it would fix the issue? | 09:56:45 |
| Arian | it’s the game “One truth and a lie” | 09:56:55 |
| Arian | lemme double-check I only tested ap-northeast-1 . Maybe they did roll out HTTP2 on us-east-1 but I’d be super surprised | 10:00:03 |
