| 16 Sep 2025 |
 shawn8901 | * I saw that there is https://github.com/NixOS/hydra/issues/1486 but my hydra.conf looks fine according to what is mentioned in the issue
❯ cat /var/lib/hydra/hydra.conf
evaluator_max_memory_size = 4096
evaluator_workers = 4
max_concurrent_evals = 1
restrict-eval = false
max_output_size = 5368709120
max_db_connections = 150
compress_build_logs = 1
<runcommand>
job = *:*:merge-pr
command = /nix/store/vfw63pq1i2r937j7aszff38msfi4d0y8-merge_pr/bin/merge_pr
</runcommand>
<hydra_notify>
<prometheus>
listen_address = 127.0.0.1
port = 9199
</prometheus>
</hydra_notify>
<githubstatus>
jobs = .*
useShortContext = true
</githubstatus>
Include /run/secrets/rendered/hydra-write-token.conf
Include /run/secrets/rendered/hydra-hook-token.conf
using_frontend_proxy = 1
base_uri = https://hydra.pointjig.de
notification_sender = hydra@pointjig.de
max_servers = 25
gc_roots_dir = /nix/var/nix/gcroots/hydra
use-substitutes = 1
so the config part of the issue seems unrelated and its more the 2nd part about the url.
I assume issue author did later just use the git url variant, but i dont think that this is a proper solution as the github url is quite nice to use.
| 19:25:24 |
| 17 Sep 2025 |
|  shift joined the room. | 16:20:11 |
| shift | Anyone care to take a look at https://github.com/NixOS/hydra/actions/runs/17803795081/job/50610337495?pr=1524 and let me know where I messed up? | 16:21:34 |
| shift | Finally a PR I authored which passes CI, and is something people have complained about for a while :D https://github.com/NixOS/hydra/pull/1525 (feat: Lazy-load Build Steps and RunCommand Logs tabs) | 17:16:52 |
| 19 Sep 2025 |
|  mkg20001 changed their profile picture. | 17:22:43 |
| 22 Sep 2025 |
 John Ericson | clear; meson test testsuite --interactive --test-args 'content-addressed/basic.t -v
| 17:16:16 |
| John Ericson | I am just gonna drop this here :) | 17:16:19 |
| John Ericson | Mic92: clear; meson test testsuite --interactive --test-args 'content-addressed/basic.t -v --keep-dirs' | 21:20:50 |
| John Ericson | --keep-dirs | 21:20:55 |
 Mic92 | John Ericson:
building ‘/tmp/yath-2368013-USwe0r/tmp/dkymXJ/yRkVF4jse4/nix/store/l1rl9139wcwrdflhfg9gzwh7pc31r70h-succeed-with-failed.drv’ on ‘ssh://localhost’
decline-permanently
| 21:31:23 |
| Mic92 | building ‘/tmp/yath-2368013-USwe0r/tmp/dkymXJ/yRkVF4jse4/nix/store/l1rl9139wcwrdflhfg9gzwh7pc31r70h-succeed-with-failed.drv’ on ‘ssh://localhost’
# decline-permanently
| 21:31:35 |
| John Ericson | hmm | 21:31:40 |
| John Ericson | sounds like the build hook | 21:31:47 |
| John Ericson | but hydra doesn't use that | 21:31:49 |
| Mic92 | * building ‘/tmp/yath-2369307-Ihb8i8/tmp/8idKEy/yRkVF4jse4/nix/store/npd67l5fmhblg3qkmilniprsrrybr3yw-non-ca-depending-on-ca.drv’ on ‘ssh://localhost’
# decline-permanently
killing process 2369672
marking build 5 as failed with buildStatus 1 (stepStatus=1, resultBuildStatus=1)
| 21:33:08 |
| Mic92 | Sandro 🐧: did you were doing ca stuff with nix 2.31 when you reported your issues? | 21:35:11 |
| Mic92 | I recored the output with sysdig -c stderr | 21:35:42 |
 Sandro | I think, yes | 21:38:28 |
| John Ericson | Sandro 🐧: going AFK, but please dump what you know here (or link if if you already wrote it) | 22:37:56 |
| Sandro | Initially it broke when updating to 2.31 but since I deployed the 2.30.4 (IIRC) update I either broke something else or it broke also. I need to see if I can revert the last or multiple flake bumps. All the builds I tried building with nix-build succeeded but even then the build by Hydra fails | 22:42:22 |
| John Ericson | ok thanks | 22:44:12 |
| John Ericson | I see the builder is killing the process thinking it's failed | 22:44:25 |
| John Ericson | (unless that is ssh killing but doubt it) | 22:44:34 |
| John Ericson | but I don't know why | 22:44:36 |
| 23 Sep 2025 |
| Sandro | It's happening with the localhost special name which shouldn't use ssh | 06:13:29 |
| Sandro | sysdig is not saying much useful
sudo sysdig -c stderr
<3>possibly transient failure building ‘/nix/store/qs7p646zcgxxrv4d9bpj3hsbapwcl8s4-source.drv’ on ‘ssh://astrid.dse.in.tum.de’:
<5>will retry ‘/nix/store/qs7p646zcgxxrv4d9bpj3hsbapwcl8s4-source.drv’ after 62s
<5>performing step ‘/nix/store/9zw0q76n5z0737bds1wpswj5q7d7ic6x-initrd-linux-6.12.47.drv’ 1 times on ‘ssh://localhost’ (needed by build 24682 and 0 others)
<5>copying 0 paths...
<5>copying 0 paths...
<3>possibly transient failure building ‘/nix/store/zphg5h6vrqj9zwx3sq0wi2dqf3f8iyfl-source.drv’ on ‘ssh://clara’:
<5>will retry ‘/nix/store/zphg5h6vrqj9zwx3sq0wi2dqf3f8iyfl-source.drv’ after 65s
accepted connection from pid 46976, user hydra-queue-runner (trusted)
<5>copying 0 paths...
<5>copying 0 paths...
created 38395 symlinks in user environment
created 38395 symlinks in user environment
<3>possibly transient failure building ‘/nix/store/sdi2gbywg0w7r0y8lvwwpl73r22j7kba-source.drv’ on ‘ssh://clara’:
<5>will retry ‘/nix/store/sdi2gbywg0w7r0y8lvwwpl73r22j7kba-source.drv’ after 66s
accepted connection from pid 47001, user hydra-queue-runner (trusted)
| 07:23:01 |
| Sandro | Sep 23 09:21:56 hydrogen hydra-queue-runner[11982]: will retry ‘/nix/store/zphg5h6vrqj9zwx3sq0wi2dqf3f8iyfl-source.drv’ after 65s
Sep 23 09:21:56 hydrogen hydra-queue-runner[11982]: copying 0 paths...
Sep 23 09:21:56 hydrogen hydra-queue-runner[11982]: copying 0 paths...
Sep 23 09:21:56 hydrogen hydra-queue-runner[11982]: possibly transient failure building ‘/nix/store/sdi2gbywg0w7r0y8lvwwpl73r22j7kba-source.drv’ on ‘ssh://clara’:
Sep 23 09:21:56 hydrogen hydra-queue-runner[11982]: will retry ‘/nix/store/sdi2gbywg0w7r0y8lvwwpl73r22j7kba-source.drv’ after 66s
Sep 23 09:21:57 hydrogen hydra-queue-runner[11982]: checking the queue for builds...
Sep 23 09:21:58 hydrogen hydra-queue-runner[11982]: copying 0 paths...
Sep 23 09:21:58 hydrogen hydra-queue-runner[11982]: copying 0 paths...
Sep 23 09:21:58 hydrogen hydra-queue-runner[11982]: marking build 24677 as failed
Sep 23 09:22:01 hydrogen hydra-queue-runner[11982]: marking build 24682 as failed
Sep 23 09:22:04 hydrogen hydra-queue-runner[11982]: possibly transient failure building ‘/nix/store/fk94qca8xwmq44h2cz4rs75sd213848k-pyccloud-0.1+20250218154744-py3-none-any.whl.drv’ on ‘ssh://astrid:
Sep 23 09:22:04 hydrogen hydra-queue-runner[11982]: will retry ‘/nix/store/fk94qca8xwmq44h2cz4rs75sd213848k-pyccloud-0.1+20250218154744-py3-none-any.whl.drv’ after 62s
Sep 23 09:22:04 hydrogen hydra-queue-runner[11982]: performing step ‘/nix/store/bqkn265gh4vnw6zmdmlrgicv9cvdsfq1-system-path.drv’ 1 times on ‘ssh://localhost’ (needed by build 24684 and 0 others)
Sep 23 09:22:04 hydrogen hydra-queue-runner[11982]: performing step ‘/nix/store/mfm7g14axg5vz1f9c7rsy5jrhwg9fsva-system-path.drv’ 1 times on ‘ssh://localhost’ (needed by build 24685 and 0 others)
Sep 23 09:22:04 hydrogen hydra-queue-runner[11982]: performing step ‘/nix/store/lkpdwpg6cvznl4vdbnyvnjya6zyvzilx-initrd-linux-6.12.47.drv’ 1 times on ‘ssh://localhost’ (needed by build 24685 and 0 others)
Sep 23 09:22:04 hydrogen hydra-queue-runner[11982]: performing step ‘/nix/store/spqafxk9w0xn5bp3ksi8lsvx7ni9sb7f-system-path.drv’ 1 times on ‘ssh://localhost’ (needed by build 24686 and 0 others)
Sep 23 09:22:04 hydrogen hydra-queue-runner[11982]: copying 0 paths...
Sep 23 09:22:04 hydrogen hydra-queue-runner[11982]: copying 0 paths...
Sep 23 09:22:04 hydrogen hydra-queue-runner[11982]: copying 0 paths...
Sep 23 09:22:04 hydrogen hydra-queue-runner[11982]: copying 0 paths...
Sep 23 09:22:04 hydrogen hydra-queue-runner[11982]: copying 0 paths...
Sep 23 09:22:04 hydrogen hydra-queue-runner[11982]: copying 0 paths...
Sep 23 09:22:05 hydrogen hydra-queue-runner[11982]: copying 0 paths...
Sep 23 09:22:05 hydrogen hydra-queue-runner[11982]: copying 0 paths...
Sep 23 09:22:06 hydrogen hydra-queue-runner[11982]: marking build 24685 as failed
Sep 23 09:22:06 hydrogen hydra-queue-runner[11982]: marking build 24684 as failed
Sep 23 09:22:07 hydrogen hydra-queue-runner[11982]: checking the queue for builds...
Sep 23 09:22:11 hydrogen hydra-queue-runner[11982]: marking build 24686 as failed
Sep 23 09:22:17 hydrogen hydra-queue-runner[11982]: checking the queue for builds...
Sep 23 09:22:27 hydrogen hydra-queue-runner[11982]: checking the queue for builds...
Sep 23 09:22:37 hydrogen hydra-queue-runner[11982]: checking the queue for builds...
Sep 23 09:22:47 hydrogen hydra-queue-runner[11982]: checking the queue for builds...
Sep 23 09:22:57 hydrogen hydra-queue-runner[11982]: checking the queue for builds...
Sep 23 09:23:07 hydrogen hydra-queue-runner[11982]: checking the queue for builds...
| 07:23:38 |
| Sandro | not sure how to provide better debugging things
Hydra 0-unstable-2025-09-13 (using nix-2.29.2+4 and nix-eval-jobs-2.31.0). You are signed in as sandro.
➜ nix --version
nix (Nix) 2.30.3+4
| 07:25:43 |
| shawn8901 | Hi, I have a short question where I did not find a good solution yet and maybe someone here had a similar "issue". I am running my own hydra instance to ci check my system configs (defined as flake) and update the flake lock file.
I'm archive that so far by defining an releaseTools.aggregate and defining a runCommand for the aggregate job. That works like a charm. As the flake is a multi arch and I have another (more) powerful machine I have defined some remote builders (which also works fine).
But sometimes the aggregate job is scheduled to one of the remote builders, thus receiving all the closures of all hosts (as far as I can tell).
Is there a way to pin the aggregate to a specific builder (best case the hydra instance?)
I tried preferLocal which seems to have no effect and a tried to pass custom systemFeatures (so that just the hydra instance should be capable of running the job), but that does not work as aggregate does not accept required systemFeatures. Does someone maybe have an idea to pin it? | 08:42:51 |
|  kenji changed their display name from a-kenji to kenji. | 10:41:21 |
