| 9 May 2026 |
 emily | we run into this issue ~every staging cycle now it feels like from what I've seen, and it's gone from "once or twice a year" to "constantly hitting users" it seems | 17:27:36 |
| emily | it also holds up cycles because it breaks downstream builds etc. | 17:28:03 |
 hexa (signing key rotation when) | yeah | 17:34:45 |
| hexa (signing key rotation when) | from may 7th 21:56 right now | 17:35:04 |
| hexa (signing key rotation when) | so less than 2 days | 17:35:15 |
| hexa (signing key rotation when) | https://termbin.com/69iy | 17:36:19 |
| hexa (signing key rotation when) | all fish related things | 17:36:22 |
| emily | yeah, not quite the retention we'd need to catch this :( | 17:38:25 |
| emily | we could check it next time we observe this cropping up in a staging-next cycle | 17:38:35 |
| emily | does S3 retain something like ^? like the date a bucket entry was added/modified? | 17:38:49 |
| emily | May 08 04:16:49 mimas hydra-queue-runner[1392977]: warning: unable to upload 'https://nix-cache.s3.us-east-1.amazonaws.com/log/xsvcvrzr8v1p7jpldddr8wkmaz84knpi-config.fish.drv': Timeout was reached (28) Connection timed out after 17368 milliseconds; retrying in 275 ms (attempt 1/5)
| 17:39:21 |
| emily | so failures definitely do happen | 17:39:29 |
| emily | (how is the connection that flaky?!) | 17:39:53 |
 Sergei Zimmerman (xokdvium) | In reply to @emilazy:matrix.org
does S3 retain something like ^? like the date a bucket entry was added/modified? Yup it’s in the headers for each file in the cache. At least I used to look at those for the frankenbuild issue in nix | 17:40:04 |
 Winter | i blame us-east-1 | 17:40:09 |
| emily | so for the fish issue:
❭ curl -i https://cache.nixos.org/62v6ki5ql5wxvgabn60aln10l2a4aacb.narinfo
HTTP/2 200
last-modified: Tue, 24 Mar 2026 04:11:52 GMT
shion:~
❭ curl -i https://cache.nixos.org/gngn7y9mn510mf1hkmr0l69qbpvxfbfh.narinfo
HTTP/2 200
last-modified: Tue, 24 Mar 2026 04:32:07 GMT
| 17:40:52 |
| emily | hexa (signing key rotation when): does 21 minutes between uploads of two outputs of the same build of a tiny package sound remotely plausible to you? | 17:41:16 |
| Sergei Zimmerman (xokdvium) | Huh, yeah this definitely got retried | 17:41:31 |
| emily | oh, the modification times are different for the NARs themselves? | 17:41:45 |
| Sergei Zimmerman (xokdvium) | How much different are we talking | 17:42:14 |
| emily | shion:~
❭ curl -i https://cache.nixos.org/nar/1fr8fly29yj9q3mznyqgrsrkrvxhh64pqv6s3smkfpwwcnmskj5m.nar.xz
HTTP/2 200
last-modified: Tue, 18 Nov 2025 01:57:28 GMT
shion:~
❭ curl -i https://cache.nixos.org/nar/135ynbv7smphw9a76x9p2849hfv85gqhmywzb5s9402nqs6fggap.nar.xz
HTTP/2 200
last-modified: Tue, 24 Mar 2026 04:32:06 GMT
| 17:42:19 |
| emily | the most differentest it's possible to be | 17:42:23 |
| emily | the old one is the -doc output | 17:42:40 |
| hexa (signing key rotation when) | We are pushing from DE to us-east-1 ... 150ms latency | 17:42:43 |
| emily | which makes sense | 17:42:45 |
| emily | it won't depend on out | 17:42:54 |
| Winter | yes, i know, hence i blame us-east-1 | 17:43:01 |
| emily | and it probably didn't change since November | 17:43:02 |
| emily | so the .narinfo files are much more canonical for this I'd imagine | 17:43:08 |
| emily | since the NARs themselves are content-addressed | 17:43:22 |
