| 18 Jun 2026 |
 hexa | uhhh, yeah. that looks off. | 11:39:21 |
| hexa | thanks for pointing that out | 11:39:31 |
 dramforever | Download bl7rmhhsy7vjb9qm3jfwgqpv3cn7wfb1.narinfo | 11:41:35 |
| dramforever | this is the correct file from TUNA
correct sha256 c2f82c7cfb1e8814fe088149bb759f614a226abd87dceae018df2893e237072c
incorrect sha256 0b66509b875567891c3f5131ad144da87a5f113446d04cbe189038ce5cf541f2 | 11:42:26 |
| dramforever | * this is the correct file from TUNA and ISCAS, both the same
correct sha256 c2f82c7cfb1e8814fe088149bb759f614a226abd87dceae018df2893e237072c
incorrect sha256 0b66509b875567891c3f5131ad144da87a5f113446d04cbe189038ce5cf541f2 | 11:42:36 |
| dramforever | the one time cache.nixos.org mirror sites are useful to upstream | 11:42:49 |
 Mic92 | I am starting to fix some keys I find in CI manually. The issue might have started this morning and the inventory is from 01:00 | 11:44:50 |
| Mic92 | * I am starting to fix some keys I find in CI manually. The issue might have started this morning and the inventory is from 01:00 UTC | 11:44:54 |
| Mic92 | So far I was able to fix errors by simply reverting the s3 object version | 11:45:21 |
| dramforever | is there any way the full list of affected objects can be published? | 11:46:09 |
| Mic92 | no not yet | 11:46:48 |
| dramforever | in the future is also fine | 11:47:02 |
| dramforever | i think | 11:47:04 |
| Mic92 | sure | 11:47:06 |
| dramforever | i noticed too late that the python script powering the TUNA and ISCAS mirrors don't actually check the FileHash (maybe because i got lazy and forgot to figure out how to deal with the traditional nix hash format...) | 11:47:31 |
| dramforever | yeah, thanks a lot | 11:47:47 |
| Mic92 | Still no new aborts observed since I deployed the queue runner fix. I restored all the broken checksums I could find so far. The proper scan will still take time. Our CI got unlocked for now by this fix: https://github.com/NixOS/infra/pull/1085 | 12:06:14 |
|  no-mood joined the room. | 12:12:45 |
 K900 | Do we know what's causing the corruption? | 12:13:06 |
| K900 | If we don't, maybe we should just stop the presses | 12:13:23 |
| Mic92 | So the queue runner was re-using deamon connections that were in a undefined state. I saw that some queue builder logs where showing that deamon queries were return garbage that must have been still there from other connections. This has likely caused that path-info queries got the wrong result | 12:17:53 |
|  magic_rb joined the room. | 12:17:56 |
| Mic92 | Now I replaced all read-only store queries by going to the db directly and open a new daemon connection for each other operation that actually requires mutation | 12:18:50 |
|  Trent Baldwin joined the room. | 12:20:16 |
|  cyborgpotato joined the room. | 12:32:51 |
|  Amboss_Mann joined the room. | 12:51:44 |
|  L14m joined the room. | 12:59:50 |
 Vladimír Čunát | Uh, while these issues are solved now probably (we had direct communication), there's some more blocking (all?) channels:
nix-support/hydra-build-products doesn't work well anymore, somehow.
| 13:13:04 |
| Vladimír Čunát | Here we're missing the "build products":
https://hydra.nixos.org/build/331662168#tabs-summary
like it was e.g. in https://hydra.nixos.org/build/331523718#tabs-summary | 13:13:40 |
| Vladimír Čunát | When I fetch the derivation, the correct file does seem to be in there. | 13:14:01 |
