!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

347 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.107 Servers

Load older messages


SenderMessageTime
8 Jul 2025
@janne.hess:helsinki-systems.de@janne.hess:helsinki-systems.de left the room.21:22:57
10 Jul 2025
@shock-wave:matrix.orgshock-wave joined the room.11:23:18
@shock-wave:matrix.orgshock-waveHi, I have been told that this chatroom is the right place to be with seek help with issues pertaining to the nixos discourse. I have been trying to log in to my account but I havent saved/remembered the password. The last time I logged in was 3 months ago. However when I try to recover my password or login with link sent to the email address, I dont receive anything. I have also tried to make a new account with a new email address but I dont seem to receive emails to that either. The account isnt banned(at least not overtly). I was wondering if someone could help me with this issue or tell me what the issue is. Thanks in advance.11:40:27
@ctheune:matrix.flyingcircus.ioChristian TheuneI can check if you DM me the email address you are using. (Room: let me know if I should verify this request further to avoid information leakage wrt potential social engineering.)11:54:37
@shock-wave:matrix.orgshock-wavewill do and understandable if you want a second opinion, I can also provide more info if needed.11:56:15
@hexa:lossy.networkhexa (signing key rotation when)infra call in 15m and I'd invite you to try lasuite-meet over here https://meet.cccda.de/rdt-xpjb-mav15:46:04
@hexa:lossy.networkhexa (signing key rotation when)* infra call in 15m and I invite you to try lasuite-meet over here https://meet.cccda.de/rdt-xpjb-mav15:46:18
@arianvp:matrix.orgArian

So we were a bit puzzled during the call.

It seems that even without my changes to cache NARs more aggressively, currently 404s are not cached even though fastly by default caches 404s.

Or at leasts x-cache-hits response header is always 0 for 404s whilst for all other paths we do get hits.

i.e. if you fetch a non-existent narinfo we always get a x-cache-hits: 0 back

So it feels like something is misconfigured today?

Or is maybe the synthetic response object that we have resetting those headers?

https://github.com/NixOS/infra/blob/main/terraform/cache.tf#L233-L240

I wonder if we need to move from cache_condition to request_condition here or something?

16:59:18
@arianvp:matrix.orgArian but yeh for me curl -v https://cache.nixos.org/non-existent.narinfo always reports a x-cache-hits: 0 which doesn't sound correct 17:00:44
@arianvp:matrix.orgArianOr does it? is that the desired behaviour?17:00:59
@fr0de_0xa:matrix.orgFred Lahde joined the room.18:47:41
@arianvp:matrix.orgArianYeh so stuff is really off: https://cache.nixos.org/h35hs85vd5nhrzv3j03ybdfz2s1wsc6l.narinfo (200) takes between 20ms and 90ms to resolve for me https://cache.nixos.org/lolno.narinfo (404) takes consistently between 120 and 230ms for me so we are not caching 404s and they're really slow18:51:27
@arianvp:matrix.orgArian

And looking at the generated VCL I indeed think we can fix it by changing cache_condition to response_condition

the cache_condition gets executed before a hit/miss is decided. This means that we return the fixed 404 response before varnish even makes decision on whether to cache or not and exits out of the VCL

18:52:29
@arianvp:matrix.orgArianso we never hit the code-path for caching18:52:33
@arianvp:matrix.orgArian"How I made `nixos-rebuild switch 200% faster for everyone with this one weird trick"18:53:14
@arianvp:matrix.orgArianI just need to know if this was deliberately set up like this. Are we on purpose not caching 404s or by accident?18:53:29
@jfly:matrix.orgJeremy Fleischman (jfly)It sure looks like the intent was to cache 404s. Here's the initial import into terraform, which has the "cache 404s" code: https://github.com/NixOS/infra/commit/ee995c5f3fee6d645a4a8fb9a93c57f3763b9f07#diff-75e932ae3525435283fff74680b6af8d8c83df93a23b10c7f0a9fcf0a6e4f3e9R179-R18418:56:46
@arianvp:matrix.orgArianyeh so it actually does the opposite. 404s are cached by default by fastly and this breaks that :D18:57:08
@jfly:matrix.orgJeremy Fleischman (jfly)i say, go delete some code18:57:40
@arianvp:matrix.orgArian well I think we still maybe want to replace the 404 payload with the string 404 otherwise we get some ugly XML blob from S3 18:58:31
@arianvp:matrix.orgArianbut we should do it later in the VCL 18:58:39
@zhaofeng:zhaofeng.liZhaofeng Liwait, do we have some kind of post-build-hook/s3 hook/etc to bust the cache after paths are built?18:59:03
@emilazy:matrix.orgemilyare you saying every Nix build in the universe is way slower than it should be because it's hitting S318:59:32
@arianvp:matrix.orgArianyes18:59:37
@emilazy:matrix.orgemily(…does S3 bill for 404s?)18:59:39
@arianvp:matrix.orgArianYes S3 bills for 404s18:59:46
@emilazy:matrix.orgemilylol18:59:49
@arianvp:matrix.orgArianthey even used to bill for authorization errors so you could just rack up anyone's bill by knowing their bucket name19:00:02
@arianvp:matrix.orgArianthey changed that now19:00:04
@emilazy:matrix.orgemilyplease run some numbers on how much of the cache size bill this is, I'm so curious19:00:13

Show newer messages


Back to Room ListRoom Version: 6