!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

350 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.108 Servers

You have reached the beginning of time (for this room).


SenderMessageTime
7 Jun 2025
@infinisil:matrix.orginfinisilAlright I'll answer with that. Can also tell them to join this room if there's other problems22:19:01
@infinisil:matrix.orginfinisil(although if they're actually blocked in some way they might not be able to :P)22:19:17
@edef1c:matrix.orgedefa lot of networking equipment also doesn't answer pings or punts them to low priority22:19:23
@edef1c:matrix.orgedefso ICMP pings are not inherently reliable tests of connectivity or latency22:19:47
@hexa:lossy.networkhexa (signing key rotation when)
In reply to @infinisil:matrix.org
(although if they're actually blocked in some way they might not be able to :P)
Unlikely
22:20:22
@edef1c:matrix.orgedef(and i can confirm that nixos.org indeed does not answer ICMP pings, but does answer HTTPS)22:20:44
@hexa:lossy.networkhexa (signing key rotation when)Relevant would he a traceroute (UDP or TCP) and a curl log22:21:08
@hexa:lossy.networkhexa (signing key rotation when) * 22:21:16
@sandro:supersandro.deSandro 🐧My providers routers drop exactly 50% of a normal mtr ping at hop 4 or 5 22:43:00
@sandro:supersandro.deSandro 🐧not 49.5, not 51.3, exactly 50.0%22:43:13
@hexa:lossy.networkhexa (signing key rotation when)its because icmp echo requests leave the fast path and go to the control plane22:43:24
@hexa:lossy.networkhexa (signing key rotation when)the control plane is not equipped to handle a huge number of packages, so rate limiting kicks in22:43:47
@sandro:supersandro.deSandro 🐧"sEcUrItY" aka make debugging hard for no reason :P or maybe they run Windows Server where you can sometimes get RCE with IPv6 pings 😂22:43:50
@hexa:lossy.networkhexa (signing key rotation when)exactly22:43:55
@hexa:lossy.networkhexa (signing key rotation when)* the control plane is not equipped to handle a huge number of packets, so rate limiting kicks in22:44:37
@sandro:supersandro.deSandro 🐧luckily you cannot turn it off for IPv6 completely without breaking some things22:44:59
@hexa:lossy.networkhexa (signing key rotation when)you absoutely can turn off icmpv6 echo requests22:45:17
@hexa:lossy.networkhexa (signing key rotation when)* you absolutely can turn off icmpv6 echo requests22:45:21
@hexa:lossy.networkhexa (signing key rotation when)but breaking neighbor discovery and path mtu discovery is where things break22:45:56
@hexa:lossy.networkhexa (signing key rotation when)* but blocking neighbor discovery and path mtu discovery is where things break22:46:00
@hexa:lossy.networkhexa (signing key rotation when)* you absolutely can turn off icmpv6 echo responses22:46:12
@hexa:lossy.networkhexa (signing key rotation when)* you absolutely can turn drop icmpv6 echo requests22:46:20
@deeok:matrix.org@deeok:matrix.org changed their display name from deeok to matrixrooms.info mod bot (does NOT read/send messages and/or invites; used for checking reported rooms).23:36:18
8 Jun 2025
@deeok:matrix.org@deeok:matrix.org left the room.00:17:57
@hexa:lossy.networkhexa (signing key rotation when)* you absolutely can drop icmpv6 echo requests15:37:33
10 Jun 2025
@arianvp:matrix.orgArian

I’m trying to understand our caching setup a bit better with Fastly<->S3, given bandwidth between S3 and fastly is our largest cost.

I noticed we’re not serving any Cache-Control headers on our NAR files, but I think we could set Cache-Control: immutable on all NAR objects in S3 as NAR files are content-addressed they should never change? Fastly respects these response headers to decide how long to cache things for.

Do I understand correctly that

https://github.com/NixOS/infra/blob/88f1c42e90ab88673ddde3bf973330fb2fcf23be/terraform/cache.tf#L138C17-L138C22

is the only thing configuring how long we hold things in cache? (seems to be 24 hours).

Given we also cache 404s on narinfos I guess that makes sense. (In case the narinfo gets uploaded later it invalidates it). But can’t we cache NARs way more aggressively than 24 hours? Would reduce the bandwidth on S3 perhaps.

11:05:09
@arianvp:matrix.orgArian *

I’m trying to understand our caching setup a bit better with Fastly<->S3, given bandwidth between S3 and fastly is our largest cost.

I noticed we’re not serving any Cache-Control headers on our NAR files, but I think we could set Cache-Control: immutable on all NAR objects in S3 as NAR files are content-addressed they should never change? Fastly respects these response headers to decide how long to cache things for.

Do I understand correctly that

https://github.com/NixOS/infra/blob/88f1c42e90ab88673ddde3bf973330fb2fcf23be/terraform/cache.tf#L138C17-L138C22

is the only thing configuring how long we hold things in cache? (seems to be 24 hours).

Given we also cache 404s on narinfos I guess that makes sense as we want them to be fast. ( and In case the narinfo gets uploaded later it invalidates it). But can’t we cache NARs way more aggressively than 24 hours? Would reduce the bandwidth on S3 perhaps.

11:06:41
@arianvp:matrix.orgArian *

I’m trying to understand our caching setup a bit better with Fastly<->S3, given bandwidth between S3 and fastly is our largest cost.

I noticed we’re not serving any Cache-Control headers on our NAR files, but I think we could set Cache-Control: immutable on all NAR objects in S3 as NAR files are content-addressed they should never change? Fastly respects these response headers it receives from S3 to decide how long to cache things for.

Do I understand correctly that

https://github.com/NixOS/infra/blob/88f1c42e90ab88673ddde3bf973330fb2fcf23be/terraform/cache.tf#L138C17-L138C22

is the only thing configuring how long we hold things in cache? (seems to be 24 hours).

Given we also cache 404s on narinfos I guess that makes sense as we want them to be fast. ( and In case the narinfo gets uploaded later it invalidates it). But can’t we cache NARs way more aggressively than 24 hours? Would reduce the bandwidth on S3 perhaps.

11:08:36

Show newer messages


Back to Room ListRoom Version: 6