| 11 May 2026 |
 John Ericson | which again is my focus right now, I tend to solve all problems with more refactoring | 19:56:54 |
| John Ericson | (we're pretty well done on the new feature work, actually) | 19:57:25 |
| John Ericson | there has been no unattended LLMs, I read code very quickly | 19:58:00 |
| John Ericson | and my knowledge of the codebase trends up not done, as I choose to take the risk of starting with the rust code I was not familiar with | 19:58:34 |
| John Ericson | we can rip out the toml formatting if you don't like that, but that is the most trivial bikeshed thing | 19:58:53 |
| John Ericson | not load-berring for the overall health of the project | 19:59:06 |
 Frooastside | Hello, I am not sure if this is the right place to ask, but I set up a hydra instance on my server and it all works so far but I am a bit confused by the architecture. I used the newer hydra-queue-builder and runner that came from helsinki-systems and when garbage collection ran on the builder, the next time it builds something, it tries to build everything starting from bootsrap stage 1. I enabled substitutions and the hydra-queue-runner says something like this:
hydra-queue-runner[84836]: 2026-05-11T19:17:16.161281Z INFO queue_monitor_loop:get_queued_builds:process_new_builds:create_build:create_step:create_step: hydra_queue_runner::state: create_step: llw11xba2z4kfiyka62yhfjc75fys75x-bash-5.3p3.drv already finished (outputs in store), skipping build_id=1 build_id=1 drv_path=a97x63dkns7qsza17rbpgkmn6k9f65sf-rp4-image-hydra.drv build_id=1 build_id=1 drv_path=a97x63dkns7qsza17rbpgkmn6k9f65sf-rp4-image-hydra.drv build_id=1 drv_path=llw11xba2z4kfiyka62yhfjc75fys75x-bash-5.3p3.drv
so it already is in the s3 cache (it got build by the builder earlier) and the queue runner knows that. the queue-builder has the cache enabled as a subsituter but doesnt use it, is this intended?
| 20:01:22 |
| John Ericson | Frooastside: this is the right spot | 20:02:49 |
| John Ericson | did you enabled substitution on the builder too? | 20:02:55 |
| John Ericson | this seems not unlike the the issues that hydra.nixos.org was facing frankly | 20:03:12 |
| John Ericson | the thing I am working on right now (but it is a massive refactor) is making the builder not get dependent drvs so it that it has to subsitute or fail, and rebuilding things is not possible | 20:03:44 |
| John Ericson | that also cuts down on traffic between the queue ruunner and the builder | 20:03:54 |
| John Ericson | the current thing is how really old remote building worked | 20:04:09 |
| John Ericson | the thing I am going to is how hydra workedf or the last decade or so | 20:05:28 |
| Frooastside | I was not sure which setting is actually required because it didnt work after just enabling services.hydra-queue-builder-dev.useSubstitutes = true; (on the builder) so I also set nix.settings.builders-use-substitutes = true (I am not sure where i found that) and this is of course also set nix.settings.substituters = ["https://cache.nixos.org" "https://cache.MY CACHE"] | 20:07:01 |
| John Ericson | yeah taht sounds like it should work | 20:08:56 |
| John Ericson | it is just hard to debug right now I am afraid with the dependency drvs being disclosed to the builder | 20:09:15 |
| John Ericson | you could try to check nix logs and figure out what it is doing what it is doing | 20:09:28 |
| John Ericson | but that is hard | 20:09:31 |
| John Ericson | (the decision to build the stuff is nix's) | 20:10:28 |
| Frooastside | This is the process running right now
hydra-q+ 84985 0.0 0.2 595328 34872 ? Sl 21:17 0:00 nix --extra-experimental-features nix-command build --json --no-pretty --print-build-logs --log-format raw-with-logs --no-link --max-silent-time 7200 --timeout 36000 --option max-build-log-size 67108864 --option fallback true --option substitute false --option builders /nix/store/bx33y6dvsb44avnjah0hsh25hj0axa0f-users-groups.json.drv^*
So it picked the first drv that is not in the cache and tries to build it but --option substitute false is set
| 20:15:15 |
| Frooastside | Thats whats confusing me, how is it supposed to work? When the builder is missing dependencies, should nix get them because substitutions should be on, should the queue-builder get them from the cache or should the queue-runner send them to the builder? | 20:21:45 |
| John Ericson | interesting, I am not sure where that is coming from | 20:34:42 |
| John Ericson | oh nevermind I do see it | 20:35:24 |
| John Ericson | that is in subprojects/crates/nix-utils/src/realise.rs | 20:37:02 |
| John Ericson | git log missed it becasue it was there since the initial commit (?) | 20:37:17 |
| John Ericson | oh because I was git loging the specific file | 20:39:10 |
| John Ericson | Frooastside: the instent is that the new builder and new queue runner will cooporate to get the inputs in place separately | 20:40:09 |
| John Ericson | so nix shouldn't need to download anything | 20:40:16 |
| John Ericson | but clearly that is not working right yet | 20:40:20 |
