| 9 May 2026 |
 emily | can any Hydra-knowers say if the sequence of events given in https://github.com/NixOS/nix/pull/15638#issuecomment-4413076030 seems at all plausible? | 17:10:20 |
| emily | I did some digging and it seems like the persistent Darwin ad-hoc code signature SIGKILL issues are indeed quite likely to be chronically caused by derivations with multiple outputs getting some of their outputs rebuilt and running into getting mangled by path rewrites | 17:10:54 |
| emily | what's not at all clear to me is why that would be happening, because any build of a derivation builds all its outputs, so as long as we have all outputs getting pushed out to the cache (rather than it being reasonably common for only some outputs to get pushed to the cache for a given build), substitutions by Hydra builders from the cache not chronically failing, and not some other weirdness like leftover outputs ending up registered in the store despite builds failing (? – recent disk space issues maybe?), I don't understand how we'd be regularly (and more commonly lately?) seeing this happen | 17:12:20 |
 hexa | wow, that's too long for me for now | 17:13:34 |
| hexa | that issue | 17:13:36 |
| emily | yeah just look at my last comment 😅 | 17:15:12 |
| emily | I can give further context as needed but the big question is just how we could end up seeing "some outputs present in the store but the derivation gets built anyway" on a regular basis on Hydra | 17:15:46 |
| emily | oh I mentioned in the previous comment before that but forgot to mention it in the second one: maybe it could also be a race condition where two builders try to build the same package, where one of them has already uploaded one output, but the second build beats it to other outputs/logs? | 17:19:51 |
| emily | the timing for that to happen seems… tight, though; I don't think fish would take long to upload… | 17:20:03 |
| emily | if there are decently logs for the cache uploads that could be accessed that would likely help narrow things down a lot | 17:21:22 |
| hexa | races are absolutely a possiblity | 17:21:47 |
| hexa | note that I tried out the new queue-runner at least three times in the last two weeks | 17:22:00 |
| emily | these issues have been present for years | 17:22:11 |
| hexa | good | 17:22:14 |
| emily | but getting worse in the past, say, couple months? | 17:22:18 |
| emily | much worse | 17:22:31 |
 Sergei Zimmerman (xokdvium) | Hm is S3 still very bad on the on queue runner? | 17:23:09 |
| emily | I think it would require
- builder A starts building
fish
- builder A finishes building
fish
- builder A uploads build log and
fish^doc
- builder B substitutes
fish^doc
- builder B starts building
fish
- builder B uploads
fish^out
- builder A doesn't get to upload
fish^out
| 17:23:56 |
| emily | seems pretty contrived to me, the entire second build has to happen between builder A starting to upload and finishing (and what would be pulling in fish^doc to begin with?) | 17:24:17 |
| hexa | nobody changed anything about s3, hydra repo has migrated to the new queue-runner i march | 17:24:41 |
| emily | seems more likely for the upload of fish^out to just hard fail the first time and then a second builder later picks it up and mangles it | 17:24:45 |
| hexa | Redacted or Malformed Event | 17:24:46 |
| hexa | the old queue-runner is gone from the repo | 17:24:52 |
| hexa | yeah, same reaction | 17:25:07 |
| hexa | I suddenly had to pin hydra without any prior communication | 17:25:18 |
| emily | do logs exist for an attempt to push out a given store path and whether it succeeded? | 17:26:11 |
| emily | or for multiple builds of a derivation that happen? like can we get data about whether fish's outputs/logs on the cache for that one derivation are actually chimerical between two separate builds? | 17:26:40 |
| Sergei Zimmerman (xokdvium) | Hm maybe last-modified could be a rough approximation? | 17:27:33 |
| emily | we run into this issue ~every staging cycle now it feels like from what I've seen, and it's gone from "once or twice a year" to "constantly hitting users" it seems | 17:27:36 |
| emily | it also holds up cycles because it breaks downstream builds etc. | 17:28:03 |
