| 19 Mar 2026 |
 hexa (signing key rotation when) | Redacted or Malformed Event | 09:55:22 |
| hexa (signing key rotation when) | This seems like an AI hallucination to me IMO
That would be wild | 09:55:52 |
| hexa (signing key rotation when) | This seems like an AI hallucination to me IMO
That would be wild | 09:55:55 |
 Arian | But the TCP-keepalive change does sound like it would fix the issue? | 09:56:45 |
| Arian | it’s the game “One truth and a lie” | 09:56:55 |
| Arian | lemme double-check I only tested ap-northeast-1 . Maybe they did roll out HTTP2 on us-east-1 but I’d be super surprised | 10:00:03 |
 Sergei Zimmerman (xokdvium) | Well if that's the case bernardo will get an earful from me since I asked him if it does and he confirmed | 10:03:36 |
| Arian | Nope 100% HTTP1.1
Host: nix-cache.s3.us-east-1.amazonaws.com
> User-Agent: curl/8.17.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
| 10:04:16 |
| Sergei Zimmerman (xokdvium) | womp womp | 10:05:00 |
| Arian | https://gist.github.com/arianvp/cf5ce0cba528acc43904d7987ae90f98 | 10:06:10 |
| Arian | You can also verify yourself with just openssl:
openssl s_client -alpn 'h2,http/1.1' -connect nix-cache.s3.amazonaws.com:443
openssl s_client -alpn 'h2' -connect nix-cache.s3.amazonaws.com:443
In the first one it negotiates HTTP1.1 and in the second one it says “No ALPN Negotiated”
| 10:12:13 |
| Sergei Zimmerman (xokdvium) | Hm so issue then would be keepalive making things worse | 10:31:54 |
 emily | shhh, SQS will hear you | 11:28:58 |
 Mic92 | I think I remember he talked about some proxies at some point. | 11:56:59 |
| Mic92 | So might be not the s3 itself | 11:57:19 |
| Mic92 | Sergei Zimmerman (xokdvium): https://github.com/NixOS/nix/pull/15522 so this would be the most sensible fix for now? | 11:57:33 |
| Arian | I feel like we have something misconfigured with curl's connection pooling | 11:58:12 |
| Mic92 | * So might be not the s3 itself that does http | 11:59:15 |
| Mic92 | * So might be not the s3 itself that does http 2.0 | 12:00:36 |
| Mic92 | hexa (signing key rotation when): so my plan, would be to apply the patch above to our hydra and if this fixes the issue, we could merge it and have a nix patch release today | 12:07:53 |
| hexa (signing key rotation when) | Works for me | 12:08:15 |
| Sergei Zimmerman (xokdvium) | Don't think so. With a dummy python server with h1.1 I do get reuse:
curl: Reusing existing http: connection with host localhost
downloading 'http://localhost:9000/9x7dq2sgrw63d93pa5lyk51hgwsmmn9k.narinfo'...
| 12:15:58 |
| Mic92 | https://github.com/NixOS/infra/pull/982 | 12:17:11 |
| Mic92 | I am checking also what the aws sdk is actually doing, since it's also using curl | 12:22:47 |
| Mic92 | https://github.com/aws/aws-sdk-cpp/blob/9204e236faaa1ca6a0342dee7caf61c7cf5ad8bb/src/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp#L172-L176 | 12:24:33 |
| Mic92 | So looks like we always had keep-alive | 12:25:12 |
| Mic92 | but aws does it's own pooling | 12:26:48 |
| Sergei Zimmerman (xokdvium) | It might have retries for the error. Also one thing to note is that old code didn't run concurrent s3 requests at all, since it was using the blocking API. Now we fire off a bunch of requests in parallel. | 12:26:53 |
| Sergei Zimmerman (xokdvium) | We do curl_multi pooling too, that does reuse the handles | 12:27:25 |
| Sergei Zimmerman (xokdvium) | Or rather the connections for the easy handles | 12:27:36 |
