| 19 Mar 2026 |
 Arian | That would be news to me. S3 has been very optimized for http 1.1 | 09:43:59 |
| Arian | Well it's the oldest Amazon service... | 09:44:03 |
 Sergei Zimmerman (xokdvium) | People did start screaming about multiple h1.1 connections:
https://github.com/NixOS/nix/issues/15208 | 09:48:42 |
 Vladimír Čunát | Those changes do sound related. | 09:49:56 |
| Arian |
arian@Arians-Mac-mini:~/ > curl https://sudachi.s3-ap-northeast-1.amazonaws.com --head
HTTP/1.1 200 OK
| 09:51:04 |
| Arian | I dont see HTTP2. And I would be extremely surprised if AWS ever will support HTTP2 on S3. | 09:52:28 |
| Arian | Yeh that seems related. We should definitely configure curl to reuse HTTP 1.1 connections. I thought it did that automatically? | 09:52:56 |
| Arian | This seems like an AI hallucination to me IMO: https://github.com/NixOS/nix/commit/695501815b0f8fb6e51ccf6598701fc459f9328f | 09:54:00 |
| Arian | This whole commit message is bogus. It’s not fixing the issue because S3 does not support HTTP2. you need to have a HTTP client that reuses connections according to the HTTP 1.1 spec | 09:54:57 |
| Vladimír Čunát |
And I would be extremely surprised if AWS ever will support HTTP2 on S3.
Perhaps skip directly to QUIC 😁
| 09:54:59 |
 hexa | Redacted or Malformed Event | 09:55:22 |
| hexa | This seems like an AI hallucination to me IMO
That would be wild | 09:55:52 |
| hexa | This seems like an AI hallucination to me IMO
That would be wild | 09:55:55 |
| Arian | But the TCP-keepalive change does sound like it would fix the issue? | 09:56:45 |
| Arian | it’s the game “One truth and a lie” | 09:56:55 |
| Arian | lemme double-check I only tested ap-northeast-1 . Maybe they did roll out HTTP2 on us-east-1 but I’d be super surprised | 10:00:03 |
| Sergei Zimmerman (xokdvium) | Well if that's the case bernardo will get an earful from me since I asked him if it does and he confirmed | 10:03:36 |
| Arian | Nope 100% HTTP1.1
Host: nix-cache.s3.us-east-1.amazonaws.com
> User-Agent: curl/8.17.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
| 10:04:16 |
| Sergei Zimmerman (xokdvium) | womp womp | 10:05:00 |
| Arian | https://gist.github.com/arianvp/cf5ce0cba528acc43904d7987ae90f98 | 10:06:10 |
| Arian | You can also verify yourself with just openssl:
openssl s_client -alpn 'h2,http/1.1' -connect nix-cache.s3.amazonaws.com:443
openssl s_client -alpn 'h2' -connect nix-cache.s3.amazonaws.com:443
In the first one it negotiates HTTP1.1 and in the second one it says “No ALPN Negotiated”
| 10:12:13 |
| Sergei Zimmerman (xokdvium) | Hm so issue then would be keepalive making things worse | 10:31:54 |
 emily | shhh, SQS will hear you | 11:28:58 |
 Mic92 | I think I remember he talked about some proxies at some point. | 11:56:59 |
| Mic92 | So might be not the s3 itself | 11:57:19 |
| Mic92 | Sergei Zimmerman (xokdvium): https://github.com/NixOS/nix/pull/15522 so this would be the most sensible fix for now? | 11:57:33 |
| Arian | I feel like we have something misconfigured with curl's connection pooling | 11:58:12 |
| Mic92 | * So might be not the s3 itself that does http | 11:59:15 |
| Mic92 | * So might be not the s3 itself that does http 2.0 | 12:00:36 |
| Mic92 | hexa (signing key rotation when): so my plan, would be to apply the patch above to our hydra and if this fixes the issue, we could merge it and have a nix patch release today | 12:07:53 |
