| 23 Sep 2025 |
 Sandro 🐧 | Just tried the following combination which I think worked earlier but no longer does...
Hydra 0-unstable-2025-09-13 (using nix-2.29.2+4 and nix-eval-jobs-2.30.0). You are signed in as sandro.
| 13:29:52 |
| Sandro 🐧 | and still the same error | 13:31:26 |
 John Ericson | @sandro:supersandro.de: ah ok thanks that's very interesting | 22:26:10 |
| John Ericson | I'll try to upgrade nix and nix eval jobs separately also | 22:26:22 |
| 24 Sep 2025 |
|  @casey:hubns.net left the room. | 15:27:19 |
| 26 Sep 2025 |
 vcunat | Hmm, weird. This command stopped working recently
curl -H "Accept: application/json" https://hydra.nixos.org/queue-runner-status
| 07:03:25 |
| vcunat | But without the accept header it does still fetch the html, but it's just harder to work with. | 07:03:56 |
| Sandro 🐧 | Seems to work for me 🤔 | 17:40:00 |
| 27 Sep 2025 |
| vcunat | Now it works for me as well. | 06:55:53 |
| 28 Sep 2025 |
| John Ericson | @joerg:thalheim.io: so I think I know how I want to debug it: | 12:17:19 |
| John Ericson | 1. Separate debug output, we just have that on Nix and keep it working in CI | 12:17:42 |
| John Ericson | 2. Make the builder sleep for a long time | 12:17:55 |
| John Ericson | Attach debugger to nix while builder is sleeping, then trace to the part where it kills the builder | 12:18:56 |
| John Ericson | * 3. Attach debugger to nix while builder is sleeping, then trace to the part where it kills the builder | 12:19:06 |
 Mic92 | Is nix a separate process in this case? | 16:08:22 |
| Mic92 | There are also tricks where a function spawns gdb to itself. Also i would use gdbstub here | 16:09:16 |
| vcunat | At a quick glance, maybe you'd consider to use rr to record an execution and then inspect that recording (which happens in a gdb interface again). | 17:16:52 |
| vcunat | * At a quick glance, maybe you'd consider using rr to record an execution and then inspect that recording (which happens in a gdb interface again). | 17:17:07 |
| John Ericson | In reply to @joerg:thalheim.io
Is nix a separate process in this case? With the ssh://localhost yes | 23:11:04 |
| John Ericson | Could use ?remote-program=... to run GDB hah | 23:11:49 |
| John Ericson | In reply to @vcunat:matrix.org
At a quick glance, maybe you'd consider using rr to record an execution and then inspect that recording (which happens in a gdb interface again). Yeah that might work well with the remote program thing | 23:12:15 |
| 29 Sep 2025 |
| Mic92 | John Ericson: we need to make the call soon. I would prefer to get rid of nix 2.29 in nixpkgs soon to not have to maintain it for another release. | 06:27:25 |
| 30 Sep 2025 |
| vcunat | Apparently it's flaky. Sometimes I get {"status":"unknown"}, sometimes a proper reply. | 09:13:26 |
 Janne | run3(['hydra-queue-runner', '--status'], \undef, \$stdout, \$stderr);
my $status;
if ($? != 0) {
$status = { status => "unknown" };
} else {
$status = decode_json($stdout);
}
| 10:58:12 |
| Janne | that happens when we cannot execute the status command | 10:58:18 |
| Janne | should be fixed with the new queue runner when we expose the metrics directly rather than going through catalyst. that would require a specific proxyPass in nginx but should simplify everything and make it more stable | 10:59:35 |
| vcunat | 🤔 so now I tried to run the command manually on that particular machine, repeatedly. Sometimes after the correct-looking JSON it added
error: queue runner did not respond; status information may be wrong
and exited with status 1.
| 11:06:41 |
| vcunat | And apparently the time to process the command fluctuates significantly. | 11:07:09 |
| vcunat | (sometimes immediate, sometimes a couple seconds) | 11:07:44 |
| vcunat | OK, I suppose it's not worth delving into details for this on the old queue runner. | 11:08:24 |
