| 9 May 2026 |
 emily | well it seems reasonable to say that Hydra giving up on an upload just never makes sense | 20:19:18 |
 hexa (signing key rotation when) | I kinda disagree | 20:19:27 |
| emily | if it gives up on uploading something to the cache, then it's just going to schedule a pointless build for it later, and then try to upload that | 20:19:29 |
| hexa (signing key rotation when) | that part is true | 20:19:44 |
| emily | which is exactly the same as continuing to try to upload, except that you do a pointless build which happens to also break things on Darwin | 20:19:46 |
| hexa (signing key rotation when) | but I also don't want an extended backlog of uploads ideally | 20:19:55 |
| emily | right, but they'll happen anyway right? | 20:20:11 |
| hexa (signing key rotation when) | we can increase the retry amounts | 20:20:12 |
| emily | they're ultimately part of the jobset | 20:20:20 |
| hexa (signing key rotation when) | except when the ydon't | 20:20:22 |
| emily | I guess the difference is it can give up on leafs | 20:20:26 |
| hexa (signing key rotation when) | huh | 20:20:28 |
| hexa (signing key rotation when) | they? | 20:20:36 |
| emily | the things being uploaded | 20:20:48 |
| hexa (signing key rotation when) | right | 20:20:52 |
| emily | I think a nicer solution is ^ where you just never push out a .narinfo for any output until all the outputs are up | 20:21:13 |
| emily | but looking at the C++ code it doesn't seem like that would be trivial to arrange if S3 can even do it | 20:21:29 |
| emily | and obviously I don't know how the new queue runner will handle uploads (maybe John Ericson does) | 20:21:44 |
| hexa (signing key rotation when) | Simon Hauser would know | 20:22:08 |
| emily | it seems like just increasing the number of retries in the Nix config specifically used by the queue runner would likely mitigate this problem in practice for now | 20:22:13 |
| hexa (signing key rotation when) | fair enough | 20:22:29 |
| hexa (signing key rotation when) | let's say 32 instead of 1024 though | 20:22:37 |
 John Ericson | I do agree with that sort of thing | 20:23:49 |
| John Ericson | (well in the CA case, it would be the build trace entry, but I digress :)) | 20:24:04 |
| emily | in the CA case the Darwin builds will be broken 100% of the time so there'll be much less debugging required to find the root cause :P | 20:24:37 |
| John Ericson | no more automatic rewriting in CA soon! :) | 20:26:12 |
| emily | how is that going to work? | 20:26:42 |
| John Ericson | https://github.com/NixOS/nix/pull/15793 see what Artemis has been working on | 20:27:51 |
| John Ericson | (note that the use of the exing protocol vs a new simpler protocol is provisional) | 20:28:07 |
| hexa (signing key rotation when) | how annoying would it be to move the S3 bucket to Europe btw 😆 | 20:28:54 |
