!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

349 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.108 Servers

Load older messages


SenderMessageTime
10 Jul 2025
@jfly:matrix.orgJeremy Fleischman (jfly)no, but the intent is to "only" cache 404s for 24 hours19:00:14
@arianvp:matrix.orgArianI have these numbers. I don't think API Calls are a large portion of our cost19:00:36
@emilazy:matrix.orgemilythat might not be great UX: the channel scripts run to bump channels after the final builds complete19:00:41
@emilazy:matrix.orgemily so anyone who has been running master before that will have cached 404s for first day everyone is bumping to the new channel revision 19:00:58
@zhaofeng:zhaofeng.liZhaofeng Licontributors can try to build locally and trigger the negative caching, which would be bad UX19:01:07
@zhaofeng:zhaofeng.liZhaofeng Liyeah19:01:10
@arianvp:matrix.orgArianBut this is already a problem. Nix caches 404s locally19:01:18
@emilazy:matrix.orgemilyit's not a problem across users19:01:25
@zhaofeng:zhaofeng.liZhaofeng Libut this would affect it for everyone19:01:29
@emilazy:matrix.orgemily most users don't try to run master, but some do 19:01:30
@arianvp:matrix.orgArianAaah I see19:01:32
@zhaofeng:zhaofeng.liZhaofeng Lisome Nix hater can even just look at the big in-progress hydra builds, and hit cache.nixos.org with the outpaths19:02:18
@emilazy:matrix.orgemilyso even someone testing a build before merging could screw up its caching for the next channel bump19:02:19
@emilazy:matrix.orgemilyso definitely needs cache-busting or to be much shorter than 24h I think19:02:31
@zhaofeng:zhaofeng.liZhaofeng Li* some Nix hater can even just look at the big in-progress hydra builds, and hit cache.nixos.org with the outpaths... then laptops everywhere will be building chromium19:03:01
@arianvp:matrix.orgArianif I'm reading the bill right we currently spend 570 dollars per month on GET requests and only 60 dollars per month on PUT PUT is charged19:08:55
@arianvp:matrix.orgArian* if I'm reading the bill right we currently spend 570 dollars per month on GET requests and only 60 dollars per month on PUT 19:09:10
@arianvp:matrix.orgArianokay so seems not caching 404s is kind of a "feature"19:14:14
@arianvp:matrix.orgArianbecause it means we don't need to do cache-busting on channel bumps19:14:39
@arianvp:matrix.orgArianbut it feels kinda annoying that the most common request is the slowest request 19:14:50
@arianvp:matrix.orgArianFound old meeting notes: https://nixos.wiki/wiki/Maintainers:Fastly 19:15:48
@arianvp:matrix.orgArian

Cache aggressive: ~1 month. (STATUS: NOT DONE -- requires upstream Hydra tooling changes, so cache uploads have their potential 404s purged in a timely manner.)

19:15:57
@emilazy:matrix.orgemilyI would guess sending a purge request would not be a very invasive Hydra change. (maybe if they need to be batched?)19:19:56
@zhaofeng:zhaofeng.liZhaofeng Liit could even be done on the S3 end too19:22:18
@arianvp:matrix.orgArianI'm not sure howe we could do this on the S3 end19:35:07
@arianvp:matrix.orgArianAh you mean: detect when something gets pushed to S3?19:35:31
@arianvp:matrix.orgArianoh yeh we could do that. We could have a Lambda or something that triggers on S3 uploads and purges the path from fastly19:36:13
@arianvp:matrix.orgArian

we do 12,078,871 PUTs per month

lambda costs 20cts per million requests. So this would cost $2.40

We could have a simple lambda trigger for each PUT and send a curl -XPURGE to the specific url in fastly

19:45:18
@emilazy:matrix.orgemilyis it billed on the fastly end?19:53:12
@arianvp:matrix.orgAriandont think so20:08:55

Show newer messages


Back to Room ListRoom Version: 6