| 9 May 2026 |
 Arian | ADHD mostly | 12:24:01 |
| Arian | I keep forgetting i was doing this | 12:24:09 |
| Arian | Only thing I'm a bit afraid of is that if we actually have people scraping old paths intelligent tiering might actually be more expensive because things get moved into more expensive tiers... But idk I think we should just apply and observe for a month. | 13:48:08 |
| Arian | Worst case is we revert | 13:48:13 |
 hexa | yeah, we can't know without trying | 13:55:45 |
| hexa | if things go to shit, what's next? gc? | 13:55:59 |
 lassulus | we do the cache exfil anyway? so maybe gc? or we get more free credits from aws. but there are multiple ideas floating around what happens if the egress costs eat up the free credits we get from amazon. would not worry too much about it for now | 13:58:00 |
| lassulus | I opushed https://github.com/NixOS/infra/pull/728 I would be happy to deploy it, but not sure if I have the right credentials :D | 13:58:29 |
| lassulus | maybe @hexa (signing key rotation when) can do that? | 13:58:33 |
| hexa | I think we can apply that during an infra call | 14:00:02 |
| hexa | the next one is on the 14th | 14:00:23 |
| lassulus | ok, we will try to be there | 14:02:15 |
| hexa | Arian does that work for you? | 14:04:47 |
| lassulus | He told me it should work. He is included in "we" :) | 14:10:03 |
 emily | can any Hydra-knowers say if the sequence of events given in https://github.com/NixOS/nix/pull/15638#issuecomment-4413076030 seems at all plausible? | 17:10:20 |
| emily | I did some digging and it seems like the persistent Darwin ad-hoc code signature SIGKILL issues are indeed quite likely to be chronically caused by derivations with multiple outputs getting some of their outputs rebuilt and running into getting mangled by path rewrites | 17:10:54 |
| emily | what's not at all clear to me is why that would be happening, because any build of a derivation builds all its outputs, so as long as we have all outputs getting pushed out to the cache (rather than it being reasonably common for only some outputs to get pushed to the cache for a given build), substitutions by Hydra builders from the cache not chronically failing, and not some other weirdness like leftover outputs ending up registered in the store despite builds failing (? – recent disk space issues maybe?), I don't understand how we'd be regularly (and more commonly lately?) seeing this happen | 17:12:20 |
| hexa | wow, that's too long for me for now | 17:13:34 |
| hexa | that issue | 17:13:36 |
| emily | yeah just look at my last comment 😅 | 17:15:12 |
| emily | I can give further context as needed but the big question is just how we could end up seeing "some outputs present in the store but the derivation gets built anyway" on a regular basis on Hydra | 17:15:46 |
| emily | oh I mentioned in the previous comment before that but forgot to mention it in the second one: maybe it could also be a race condition where two builders try to build the same package, where one of them has already uploaded one output, but the second build beats it to other outputs/logs? | 17:19:51 |
| emily | the timing for that to happen seems… tight, though; I don't think fish would take long to upload… | 17:20:03 |
| emily | if there are decently logs for the cache uploads that could be accessed that would likely help narrow things down a lot | 17:21:22 |
| hexa | races are absolutely a possiblity | 17:21:47 |
| hexa | note that I tried out the new queue-runner at least three times in the last two weeks | 17:22:00 |
| emily | these issues have been present for years | 17:22:11 |
| hexa | good | 17:22:14 |
| emily | but getting worse in the past, say, couple months? | 17:22:18 |
| emily | much worse | 17:22:31 |
