| 18 Jun 2026 |
 Mic92 | I have now a list of 8k nars to check. The inventory is only a snapshot though, so there might be some store paths from today that I have to check when the new snapshot comes in | 11:15:27 |
 dramforever | so um, i think it's not just nar files that are affected, like this openssl which looks very wrong
cc hexa (signing key rotation when)
$ curl https://cache.nixos.org/bl7rmhhsy7vjb9qm3jfwgqpv3cn7wfb1.narinfo
StorePath: /nix/store/bl7rmhhsy7vjb9qm3jfwgqpv3cn7wfb1-openssl-3.6.2
URL: nar/1fypgxjwksn6kh2ig4fmaskfnflicjq60zf5il2s9nxblhkkvnww.nar.zst
Compression: zstd
FileHash: sha256:0c2mjp3jk671s6ld10x93g95sdhciljnkb8bngdyxn1lj3rkc5hj
FileSize: 3777677
NarHash: sha256:0nagwij5x8yi54380yw53xy1ksn9jclp98lxmizbqq6hgngqp553
NarSize: 148512
Deriver: 2kabrm3sjvrax28i9n5wl7hzrlf3q8dg-sudo-1.9.17p2.drv
Sig: cache.nixos.org-1:NUIiNu9yrFL1630mONHKpI/av4U7kxUXVgVOWlVZqUo0VOUbMTaBPycP8jLvbIYkNHxAVJ7bpp41HaqJAB7VCw==
| 11:37:32 |
 hexa | can you point out what you're seeing? | 11:37:59 |
| dramforever | Deriver: 2kabrm3sjvrax28i9n5wl7hzrlf3q8dg-sudo-1.9.17p2.drv, and no References | 11:38:14 |
| hexa | uhhh, yeah. that looks off. | 11:39:21 |
| hexa | thanks for pointing that out | 11:39:31 |
| dramforever | Download bl7rmhhsy7vjb9qm3jfwgqpv3cn7wfb1.narinfo | 11:41:35 |
| dramforever | this is the correct file from TUNA
correct sha256 c2f82c7cfb1e8814fe088149bb759f614a226abd87dceae018df2893e237072c
incorrect sha256 0b66509b875567891c3f5131ad144da87a5f113446d04cbe189038ce5cf541f2 | 11:42:26 |
| dramforever | * this is the correct file from TUNA and ISCAS, both the same
correct sha256 c2f82c7cfb1e8814fe088149bb759f614a226abd87dceae018df2893e237072c
incorrect sha256 0b66509b875567891c3f5131ad144da87a5f113446d04cbe189038ce5cf541f2 | 11:42:36 |
| dramforever | the one time cache.nixos.org mirror sites are useful to upstream | 11:42:49 |
| Mic92 | I am starting to fix some keys I find in CI manually. The issue might have started this morning and the inventory is from 01:00 | 11:44:50 |
| Mic92 | * I am starting to fix some keys I find in CI manually. The issue might have started this morning and the inventory is from 01:00 UTC | 11:44:54 |
| Mic92 | So far I was able to fix errors by simply reverting the s3 object version | 11:45:21 |
| dramforever | is there any way the full list of affected objects can be published? | 11:46:09 |
| Mic92 | no not yet | 11:46:48 |
| dramforever | in the future is also fine | 11:47:02 |
| dramforever | i think | 11:47:04 |
| Mic92 | sure | 11:47:06 |
| dramforever | i noticed too late that the python script powering the TUNA and ISCAS mirrors don't actually check the FileHash (maybe because i got lazy and forgot to figure out how to deal with the traditional nix hash format...) | 11:47:31 |
| dramforever | yeah, thanks a lot | 11:47:47 |
| Mic92 | Still no new aborts observed since I deployed the queue runner fix. I restored all the broken checksums I could find so far. The proper scan will still take time. Our CI got unlocked for now by this fix: https://github.com/NixOS/infra/pull/1085 | 12:06:14 |
|  no-mood joined the room. | 12:12:45 |
 K900 | Do we know what's causing the corruption? | 12:13:06 |
| K900 | If we don't, maybe we should just stop the presses | 12:13:23 |
| Mic92 | So the queue runner was re-using deamon connections that were in a undefined state. I saw that some queue builder logs where showing that deamon queries were return garbage that must have been still there from other connections. This has likely caused that path-info queries got the wrong result | 12:17:53 |
|  magic_rb joined the room. | 12:17:56 |
| Mic92 | Now I replaced all read-only store queries by going to the db directly and open a new daemon connection for each other operation that actually requires mutation | 12:18:50 |
|  Trent Baldwin joined the room. | 12:20:16 |
|  cyborgpotato joined the room. | 12:32:51 |
|  Amboss_Mann joined the room. | 12:51:44 |
