| 18 Jun 2026 |
 Mic92 | I am scanning the cache | 10:30:09 |
| Mic92 | To find all candidates | 10:30:15 |
 Vladimír Čunát | I keep hating that we have to "build" all constituents to pass the job (they can be huge closures), but that's mostly an independent issue. | 10:30:38 |
| Mic92 | Just checking the version won't be enough if it also affects new store path | 10:30:47 |
| Mic92 | * Just checking the bucket version won't be enough if it also affects new store path | 10:30:59 |
| Mic92 | It will help with recovery a bit though | 10:31:07 |
| Vladimír Čunát | * I keep hating that we have to "build" by moving all constituents to a builder - in order to pass the job (they can be huge closures), but that's mostly an independent issue. | 10:31:05 |
| Vladimír Čunát | Non-FOD derivations should get obsolete within a month or two. | 10:32:20 |
| Vladimír Čunát | * Non-FOD derivations should get "obsolete" within a month or two. | 10:32:44 |
| Vladimír Čunát | (because of staging rebuilds) | 10:32:52 |
| Vladimír Čunát | * Almost all non-FOD derivations should get "obsolete" within a month or two. | 10:33:08 |
| Mic92 | vcunat: didn't see more aborts that was caused by the issue I was trying to fix. will keep watching | 10:40:19 |
|  Ben Sparks joined the room. | 10:48:58 |
|  eyJhb joined the room. | 10:54:53 |
| Mic92 | No new aborts. I am now using the s3 inventory as Arian suggested to get all the narinfos from this point in time | 10:59:08 |
|  same joined the room. | 11:02:01 |
| Mic92 | I have now a list of 8k nars to check. The inventory is only a snapshot though, so there might be some store paths from today that I have to check when the new snapshot comes in | 11:15:27 |
 dramforever | so um, i think it's not just nar files that are affected, like this openssl which looks very wrong
cc hexa (signing key rotation when)
$ curl https://cache.nixos.org/bl7rmhhsy7vjb9qm3jfwgqpv3cn7wfb1.narinfo
StorePath: /nix/store/bl7rmhhsy7vjb9qm3jfwgqpv3cn7wfb1-openssl-3.6.2
URL: nar/1fypgxjwksn6kh2ig4fmaskfnflicjq60zf5il2s9nxblhkkvnww.nar.zst
Compression: zstd
FileHash: sha256:0c2mjp3jk671s6ld10x93g95sdhciljnkb8bngdyxn1lj3rkc5hj
FileSize: 3777677
NarHash: sha256:0nagwij5x8yi54380yw53xy1ksn9jclp98lxmizbqq6hgngqp553
NarSize: 148512
Deriver: 2kabrm3sjvrax28i9n5wl7hzrlf3q8dg-sudo-1.9.17p2.drv
Sig: cache.nixos.org-1:NUIiNu9yrFL1630mONHKpI/av4U7kxUXVgVOWlVZqUo0VOUbMTaBPycP8jLvbIYkNHxAVJ7bpp41HaqJAB7VCw==
| 11:37:32 |
 hexa | can you point out what you're seeing? | 11:37:59 |
| dramforever | Deriver: 2kabrm3sjvrax28i9n5wl7hzrlf3q8dg-sudo-1.9.17p2.drv, and no References | 11:38:14 |
| hexa | uhhh, yeah. that looks off. | 11:39:21 |
| hexa | thanks for pointing that out | 11:39:31 |
| dramforever | Download bl7rmhhsy7vjb9qm3jfwgqpv3cn7wfb1.narinfo | 11:41:35 |
| dramforever | this is the correct file from TUNA
correct sha256 c2f82c7cfb1e8814fe088149bb759f614a226abd87dceae018df2893e237072c
incorrect sha256 0b66509b875567891c3f5131ad144da87a5f113446d04cbe189038ce5cf541f2 | 11:42:26 |
| dramforever | * this is the correct file from TUNA and ISCAS, both the same
correct sha256 c2f82c7cfb1e8814fe088149bb759f614a226abd87dceae018df2893e237072c
incorrect sha256 0b66509b875567891c3f5131ad144da87a5f113446d04cbe189038ce5cf541f2 | 11:42:36 |
| dramforever | the one time cache.nixos.org mirror sites are useful to upstream | 11:42:49 |
| Mic92 | I am starting to fix some keys I find in CI manually. The issue might have started this morning and the inventory is from 01:00 | 11:44:50 |
| Mic92 | * I am starting to fix some keys I find in CI manually. The issue might have started this morning and the inventory is from 01:00 UTC | 11:44:54 |
| Mic92 | So far I was able to fix errors by simply reverting the s3 object version | 11:45:21 |
| dramforever | is there any way the full list of affected objects can be published? | 11:46:09 |
