| 7 Jun 2025 |
 infinisil | (although if they're actually blocked in some way they might not be able to :P) | 22:19:17 |
 edef | a lot of networking equipment also doesn't answer pings or punts them to low priority | 22:19:23 |
| edef | so ICMP pings are not inherently reliable tests of connectivity or latency | 22:19:47 |
 hexa | In reply to @infinisil:matrix.org
(although if they're actually blocked in some way they might not be able to :P) Unlikely | 22:20:22 |
| edef | (and i can confirm that nixos.org indeed does not answer ICMP pings, but does answer HTTPS) | 22:20:44 |
| hexa | Relevant would he a traceroute (UDP or TCP) and a curl log | 22:21:08 |
| hexa | * | 22:21:16 |
 Sandro | My providers routers drop exactly 50% of a normal mtr ping at hop 4 or 5 | 22:43:00 |
| Sandro | not 49.5, not 51.3, exactly 50.0% | 22:43:13 |
| hexa | its because icmp echo requests leave the fast path and go to the control plane | 22:43:24 |
| hexa | the control plane is not equipped to handle a huge number of packages, so rate limiting kicks in | 22:43:47 |
| Sandro | "sEcUrItY" aka make debugging hard for no reason :P
or maybe they run Windows Server where you can sometimes get RCE with IPv6 pings 😂 | 22:43:50 |
| hexa | exactly | 22:43:55 |
| hexa | * the control plane is not equipped to handle a huge number of packets, so rate limiting kicks in | 22:44:37 |
| Sandro | luckily you cannot turn it off for IPv6 completely without breaking some things | 22:44:59 |
| hexa | you absoutely can turn off icmpv6 echo requests | 22:45:17 |
| hexa | * you absolutely can turn off icmpv6 echo requests | 22:45:21 |
| hexa | but breaking neighbor discovery and path mtu discovery is where things break | 22:45:56 |
| hexa | * but blocking neighbor discovery and path mtu discovery is where things break | 22:46:00 |
| hexa | * you absolutely can turn off icmpv6 echo responses | 22:46:12 |
| hexa | * you absolutely can turn drop icmpv6 echo requests | 22:46:20 |
|  @deeok:matrix.org changed their display name from deeok to matrixrooms.info mod bot (does NOT read/send messages and/or invites; used for checking reported rooms). | 23:36:18 |
| 8 Jun 2025 |
| @deeok:matrix.org left the room. | 00:17:57 |
| hexa | * you absolutely can drop icmpv6 echo requests | 15:37:33 |
| 10 Jun 2025 |
 Arian | I’m trying to understand our caching setup a bit better with Fastly<->S3, given bandwidth between S3 and fastly is our largest cost.
I noticed we’re not serving any Cache-Control headers on our NAR files, but I think we could set Cache-Control: immutable on all NAR objects in S3 as NAR files are content-addressed they should never change? Fastly respects these response headers to decide how long to cache things for.
Do I understand correctly that
https://github.com/NixOS/infra/blob/88f1c42e90ab88673ddde3bf973330fb2fcf23be/terraform/cache.tf#L138C17-L138C22
is the only thing configuring how long we hold things in cache? (seems to be 24 hours).
Given we also cache 404s on narinfos I guess that makes sense. (In case the narinfo gets uploaded later it invalidates it). But can’t we cache NARs way more aggressively than 24 hours? Would reduce the bandwidth on S3 perhaps.
| 11:05:09 |
| Arian | * I’m trying to understand our caching setup a bit better with Fastly<->S3, given bandwidth between S3 and fastly is our largest cost.
I noticed we’re not serving any Cache-Control headers on our NAR files, but I think we could set Cache-Control: immutable on all NAR objects in S3 as NAR files are content-addressed they should never change? Fastly respects these response headers to decide how long to cache things for.
Do I understand correctly that
https://github.com/NixOS/infra/blob/88f1c42e90ab88673ddde3bf973330fb2fcf23be/terraform/cache.tf#L138C17-L138C22
is the only thing configuring how long we hold things in cache? (seems to be 24 hours).
Given we also cache 404s on narinfos I guess that makes sense as we want them to be fast. ( and In case the narinfo gets uploaded later it invalidates it). But can’t we cache NARs way more aggressively than 24 hours? Would reduce the bandwidth on S3 perhaps.
| 11:06:41 |
| Arian | * I’m trying to understand our caching setup a bit better with Fastly<->S3, given bandwidth between S3 and fastly is our largest cost.
I noticed we’re not serving any Cache-Control headers on our NAR files, but I think we could set Cache-Control: immutable on all NAR objects in S3 as NAR files are content-addressed they should never change? Fastly respects these response headers it receives from S3 to decide how long to cache things for.
Do I understand correctly that
https://github.com/NixOS/infra/blob/88f1c42e90ab88673ddde3bf973330fb2fcf23be/terraform/cache.tf#L138C17-L138C22
is the only thing configuring how long we hold things in cache? (seems to be 24 hours).
Given we also cache 404s on narinfos I guess that makes sense as we want them to be fast. ( and In case the narinfo gets uploaded later it invalidates it). But can’t we cache NARs way more aggressively than 24 hours? Would reduce the bandwidth on S3 perhaps.
| 11:08:36 |
| Arian | I guess even for 200 OK narinfos we could set Cache-Control: immutable. Just not for 404s | 11:11:06 |
 emily | FWIW I don't know how Fastly's cache expiration works but it's possible that longer caching could make things meaningfully faster too. I've noticed that fetching an ISO NAR from the cache goes at ~500 Mbit/s the first time and then maxes out my connection after that (last I checked the HTTP headers imply that it's already cached on Fastly but just not at my edge location but they don't seem to update right so I'm not sure if I should trust that) | 11:18:29 |
