| 7 Jun 2025 |
 edef | so ICMP pings are not inherently reliable tests of connectivity or latency | 22:19:47 |
 hexa (signing key rotation when) | In reply to @infinisil:matrix.org
(although if they're actually blocked in some way they might not be able to :P) Unlikely | 22:20:22 |
| edef | (and i can confirm that nixos.org indeed does not answer ICMP pings, but does answer HTTPS) | 22:20:44 |
| hexa (signing key rotation when) | Relevant would he a traceroute (UDP or TCP) and a curl log | 22:21:08 |
| hexa (signing key rotation when) | * | 22:21:16 |
 Sandro 🐧 | My providers routers drop exactly 50% of a normal mtr ping at hop 4 or 5 | 22:43:00 |
| Sandro 🐧 | not 49.5, not 51.3, exactly 50.0% | 22:43:13 |
| hexa (signing key rotation when) | its because icmp echo requests leave the fast path and go to the control plane | 22:43:24 |
| hexa (signing key rotation when) | the control plane is not equipped to handle a huge number of packages, so rate limiting kicks in | 22:43:47 |
| Sandro 🐧 | "sEcUrItY" aka make debugging hard for no reason :P
or maybe they run Windows Server where you can sometimes get RCE with IPv6 pings 😂 | 22:43:50 |
| hexa (signing key rotation when) | exactly | 22:43:55 |
| hexa (signing key rotation when) | * the control plane is not equipped to handle a huge number of packets, so rate limiting kicks in | 22:44:37 |
| Sandro 🐧 | luckily you cannot turn it off for IPv6 completely without breaking some things | 22:44:59 |
| hexa (signing key rotation when) | you absoutely can turn off icmpv6 echo requests | 22:45:17 |
| hexa (signing key rotation when) | * you absolutely can turn off icmpv6 echo requests | 22:45:21 |
| hexa (signing key rotation when) | but breaking neighbor discovery and path mtu discovery is where things break | 22:45:56 |
| hexa (signing key rotation when) | * but blocking neighbor discovery and path mtu discovery is where things break | 22:46:00 |
| hexa (signing key rotation when) | * you absolutely can turn off icmpv6 echo responses | 22:46:12 |
| hexa (signing key rotation when) | * you absolutely can turn drop icmpv6 echo requests | 22:46:20 |
|  @deeok:matrix.org changed their display name from deeok to matrixrooms.info mod bot (does NOT read/send messages and/or invites; used for checking reported rooms). | 23:36:18 |
| 8 Jun 2025 |
| @deeok:matrix.org left the room. | 00:17:57 |
| hexa (signing key rotation when) | * you absolutely can drop icmpv6 echo requests | 15:37:33 |
| 10 Jun 2025 |
 Arian | I’m trying to understand our caching setup a bit better with Fastly<->S3, given bandwidth between S3 and fastly is our largest cost.
I noticed we’re not serving any Cache-Control headers on our NAR files, but I think we could set Cache-Control: immutable on all NAR objects in S3 as NAR files are content-addressed they should never change? Fastly respects these response headers to decide how long to cache things for.
Do I understand correctly that
https://github.com/NixOS/infra/blob/88f1c42e90ab88673ddde3bf973330fb2fcf23be/terraform/cache.tf#L138C17-L138C22
is the only thing configuring how long we hold things in cache? (seems to be 24 hours).
Given we also cache 404s on narinfos I guess that makes sense. (In case the narinfo gets uploaded later it invalidates it). But can’t we cache NARs way more aggressively than 24 hours? Would reduce the bandwidth on S3 perhaps.
| 11:05:09 |
| Arian | * I’m trying to understand our caching setup a bit better with Fastly<->S3, given bandwidth between S3 and fastly is our largest cost.
I noticed we’re not serving any Cache-Control headers on our NAR files, but I think we could set Cache-Control: immutable on all NAR objects in S3 as NAR files are content-addressed they should never change? Fastly respects these response headers to decide how long to cache things for.
Do I understand correctly that
https://github.com/NixOS/infra/blob/88f1c42e90ab88673ddde3bf973330fb2fcf23be/terraform/cache.tf#L138C17-L138C22
is the only thing configuring how long we hold things in cache? (seems to be 24 hours).
Given we also cache 404s on narinfos I guess that makes sense as we want them to be fast. ( and In case the narinfo gets uploaded later it invalidates it). But can’t we cache NARs way more aggressively than 24 hours? Would reduce the bandwidth on S3 perhaps.
| 11:06:41 |
| Arian | * I’m trying to understand our caching setup a bit better with Fastly<->S3, given bandwidth between S3 and fastly is our largest cost.
I noticed we’re not serving any Cache-Control headers on our NAR files, but I think we could set Cache-Control: immutable on all NAR objects in S3 as NAR files are content-addressed they should never change? Fastly respects these response headers it receives from S3 to decide how long to cache things for.
Do I understand correctly that
https://github.com/NixOS/infra/blob/88f1c42e90ab88673ddde3bf973330fb2fcf23be/terraform/cache.tf#L138C17-L138C22
is the only thing configuring how long we hold things in cache? (seems to be 24 hours).
Given we also cache 404s on narinfos I guess that makes sense as we want them to be fast. ( and In case the narinfo gets uploaded later it invalidates it). But can’t we cache NARs way more aggressively than 24 hours? Would reduce the bandwidth on S3 perhaps.
| 11:08:36 |
| Arian | I guess even for 200 OK narinfos we could set Cache-Control: immutable. Just not for 404s | 11:11:06 |
 emily | FWIW I don't know how Fastly's cache expiration works but it's possible that longer caching could make things meaningfully faster too. I've noticed that fetching an ISO NAR from the cache goes at ~500 Mbit/s the first time and then maxes out my connection after that (last I checked the HTTP headers imply that it's already cached on Fastly but just not at my edge location but they don't seem to update right so I'm not sure if I should trust that) | 11:18:29 |
| emily | no idea if the cap is Fastly–Fastly or Fastly–S3 or what, but just throwing it out there | 11:19:14 |
| Arian | IDK if S3 supports setting Cache-Control: immutable on objects. But it for sure does support Cache-Control: max-age=XXX. We could also override the TTL for /nar path in VCL to increase the max-age to the maximum value that Fastly supports (seems to be a year) | 11:28:32 |
| Arian | * IDK if S3 supports setting Cache-Control: immutable on objects. But it for sure does support Cache-Control: max-age=XXX. We could also override the TTL for /nar path in VCL to increase the max-age to the maximum value that Fastly supports (seems to be a year). Because setting this on the S3 level would require changes to Nix to set those as request headers when uploading To s3. | 11:29:02 |
