| 19 Mar 2026 |
 Arian | I dont see HTTP2. And I would be extremely surprised if AWS ever will support HTTP2 on S3. | 09:52:28 |
| Arian | Yeh that seems related. We should definitely configure curl to reuse HTTP 1.1 connections. I thought it did that automatically? | 09:52:56 |
| Arian | This seems like an AI hallucination to me IMO: https://github.com/NixOS/nix/commit/695501815b0f8fb6e51ccf6598701fc459f9328f | 09:54:00 |
| Arian | This whole commit message is bogus. It’s not fixing the issue because S3 does not support HTTP2. you need to have a HTTP client that reuses connections according to the HTTP 1.1 spec | 09:54:57 |
 Vladimír Čunát |
And I would be extremely surprised if AWS ever will support HTTP2 on S3.
Perhaps skip directly to QUIC 😁
| 09:54:59 |
 hexa | Redacted or Malformed Event | 09:55:22 |
| hexa | This seems like an AI hallucination to me IMO
That would be wild | 09:55:52 |
| hexa | This seems like an AI hallucination to me IMO
That would be wild | 09:55:55 |
| Arian | But the TCP-keepalive change does sound like it would fix the issue? | 09:56:45 |
| Arian | it’s the game “One truth and a lie” | 09:56:55 |
| Arian | lemme double-check I only tested ap-northeast-1 . Maybe they did roll out HTTP2 on us-east-1 but I’d be super surprised | 10:00:03 |
 Sergei Zimmerman (xokdvium) | Well if that's the case bernardo will get an earful from me since I asked him if it does and he confirmed | 10:03:36 |
| Arian | Nope 100% HTTP1.1
Host: nix-cache.s3.us-east-1.amazonaws.com
> User-Agent: curl/8.17.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
| 10:04:16 |
| Sergei Zimmerman (xokdvium) | womp womp | 10:05:00 |
| Arian | https://gist.github.com/arianvp/cf5ce0cba528acc43904d7987ae90f98 | 10:06:10 |
| Arian | You can also verify yourself with just openssl:
openssl s_client -alpn 'h2,http/1.1' -connect nix-cache.s3.amazonaws.com:443
openssl s_client -alpn 'h2' -connect nix-cache.s3.amazonaws.com:443
In the first one it negotiates HTTP1.1 and in the second one it says “No ALPN Negotiated”
| 10:12:13 |
| Sergei Zimmerman (xokdvium) | Hm so issue then would be keepalive making things worse | 10:31:54 |
 emily | shhh, SQS will hear you | 11:28:58 |
 Mic92 | I think I remember he talked about some proxies at some point. | 11:56:59 |
| Mic92 | So might be not the s3 itself | 11:57:19 |
| Mic92 | Sergei Zimmerman (xokdvium): https://github.com/NixOS/nix/pull/15522 so this would be the most sensible fix for now? | 11:57:33 |
| Arian | I feel like we have something misconfigured with curl's connection pooling | 11:58:12 |
| Mic92 | * So might be not the s3 itself that does http | 11:59:15 |
| Mic92 | * So might be not the s3 itself that does http 2.0 | 12:00:36 |
| Mic92 | hexa (signing key rotation when): so my plan, would be to apply the patch above to our hydra and if this fixes the issue, we could merge it and have a nix patch release today | 12:07:53 |
| hexa | Works for me | 12:08:15 |
| Sergei Zimmerman (xokdvium) | Don't think so. With a dummy python server with h1.1 I do get reuse:
curl: Reusing existing http: connection with host localhost
downloading 'http://localhost:9000/9x7dq2sgrw63d93pa5lyk51hgwsmmn9k.narinfo'...
| 12:15:58 |
| Mic92 | https://github.com/NixOS/infra/pull/982 | 12:17:11 |
| Mic92 | I am checking also what the aws sdk is actually doing, since it's also using curl | 12:22:47 |
| Mic92 | https://github.com/aws/aws-sdk-cpp/blob/9204e236faaa1ca6a0342dee7caf61c7cf5ad8bb/src/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp#L172-L176 | 12:24:33 |
