| 9 May 2026 |
 hexa | from may 7th 21:56 right now | 17:35:04 |
| hexa | so less than 2 days | 17:35:15 |
| hexa | https://termbin.com/69iy | 17:36:19 |
| hexa | all fish related things | 17:36:22 |
 emily | yeah, not quite the retention we'd need to catch this :( | 17:38:25 |
| emily | we could check it next time we observe this cropping up in a staging-next cycle | 17:38:35 |
| emily | does S3 retain something like ^? like the date a bucket entry was added/modified? | 17:38:49 |
| emily | May 08 04:16:49 mimas hydra-queue-runner[1392977]: warning: unable to upload 'https://nix-cache.s3.us-east-1.amazonaws.com/log/xsvcvrzr8v1p7jpldddr8wkmaz84knpi-config.fish.drv': Timeout was reached (28) Connection timed out after 17368 milliseconds; retrying in 275 ms (attempt 1/5)
| 17:39:21 |
| emily | so failures definitely do happen | 17:39:29 |
| emily | (how is the connection that flaky?!) | 17:39:53 |
 Sergei Zimmerman (xokdvium) | In reply to @emilazy:matrix.org
does S3 retain something like ^? like the date a bucket entry was added/modified? Yup it’s in the headers for each file in the cache. At least I used to look at those for the frankenbuild issue in nix | 17:40:04 |
 Winter | i blame us-east-1 | 17:40:09 |
| emily | so for the fish issue:
❭ curl -i https://cache.nixos.org/62v6ki5ql5wxvgabn60aln10l2a4aacb.narinfo
HTTP/2 200
last-modified: Tue, 24 Mar 2026 04:11:52 GMT
shion:~
❭ curl -i https://cache.nixos.org/gngn7y9mn510mf1hkmr0l69qbpvxfbfh.narinfo
HTTP/2 200
last-modified: Tue, 24 Mar 2026 04:32:07 GMT
| 17:40:52 |
| emily | hexa (signing key rotation when): does 21 minutes between uploads of two outputs of the same build of a tiny package sound remotely plausible to you? | 17:41:16 |
| Sergei Zimmerman (xokdvium) | Huh, yeah this definitely got retried | 17:41:31 |
| emily | oh, the modification times are different for the NARs themselves? | 17:41:45 |
| Sergei Zimmerman (xokdvium) | How much different are we talking | 17:42:14 |
| emily | shion:~
❭ curl -i https://cache.nixos.org/nar/1fr8fly29yj9q3mznyqgrsrkrvxhh64pqv6s3smkfpwwcnmskj5m.nar.xz
HTTP/2 200
last-modified: Tue, 18 Nov 2025 01:57:28 GMT
shion:~
❭ curl -i https://cache.nixos.org/nar/135ynbv7smphw9a76x9p2849hfv85gqhmywzb5s9402nqs6fggap.nar.xz
HTTP/2 200
last-modified: Tue, 24 Mar 2026 04:32:06 GMT
| 17:42:19 |
| emily | the most differentest it's possible to be | 17:42:23 |
| emily | the old one is the -doc output | 17:42:40 |
| hexa | We are pushing from DE to us-east-1 ... 150ms latency | 17:42:43 |
| emily | which makes sense | 17:42:45 |
| emily | it won't depend on out | 17:42:54 |
| Winter | yes, i know, hence i blame us-east-1 | 17:43:01 |
| emily | and it probably didn't change since November | 17:43:02 |
| emily | so the .narinfo files are much more canonical for this I'd imagine | 17:43:08 |
| emily | since the NARs themselves are content-addressed | 17:43:22 |
| Sergei Zimmerman (xokdvium) | Yeah, narinfo is the one to look out for then | 17:43:40 |
| emily | basically if it's on the download end then we could boost the substitution timeouts or number of attempts. if it's on the upload end then we could boost the upload timeouts or number of attempts. in both cases it'd be quite nice if we could rule out the partial case altogether of course. but that might be hard (e.g. can we make S3 atomically publish multiple .narinfo files? totally unclear to me) | 17:44:57 |
| emily | oh bingo | 17:46:22 |
