| 26 Jun 2026 |
 hexa (signing key rotation when) | but the process is actually running? | 08:51:44 |
| hexa (signing key rotation when) | # strace -fp 1061238
strace: Process 1061238 attached
restart_syscall(<... resuming interrupted restart_syscall ...>
| 08:51:58 |
 vcunat | It started about 5h ago and produced an eval. | 08:53:08 |
| hexa (signing key rotation when) | so it does not exit properly anymore and blocks new evals? | 08:53:34 |
| hexa (signing key rotation when) | killed the job | 08:53:58 |
| hexa (signing key rotation when) | it will certainly return | 08:54:08 |
| vcunat | Uh, log for that service only has the last 2 hours. | 08:54:11 |
| hexa (signing key rotation when) | yeah, queue-runner spam | 08:54:19 |
| hexa (signing key rotation when) | and journal being limited at 4.0G right now | 08:54:37 |
| vcunat | I'm quite sure that Hydra was showing the "running" status even after finishing and eval and then finished some other evals, too. | 08:55:44 |
| vcunat | (and it still showed the running status on the first one) | 08:55:56 |
| vcunat | * I'm quite sure that Hydra was showing the "running" status even after finishing an eval and then finished some other evals, too. | 08:56:08 |
| hexa (signing key rotation when) | after killing it, it started evaling unstable-small | 08:56:18 |
 Mic92 | I did remove some spamy messages. But I think I need to do another round | 08:56:24 |
| Mic92 | some log fields are also print as bytes still | 08:56:33 |
| vcunat | Either way, this weird behavior is a new thing. | 08:56:34 |
| vcunat | And I've restarted the service at least once after appearing stuck, sometime in the last 3 days. | 08:57:31 |
| hexa (signing key rotation when) | right now we log every build step on INFO | 08:59:02 |
| hexa (signing key rotation when) | that already creates a huge volume of logs | 08:59:10 |
| hexa (signing key rotation when) | Redacted or Malformed Event | 08:59:13 |
| vcunat | I expect it's useful now when fixing new queue-runner bugs. | 09:00:49 |
| vcunat | Maybe we'd better increase the overall journal size limit? | 09:01:14 |
| vcunat | In this case, a few GBs shouldn't matter. | 09:02:41 |
| vcunat | (I see 831G available for / right now.) | 09:03:25 |
| hexa (signing key rotation when) | looks like pgmpy | 09:57:30 |
| hexa (signing key rotation when) | because of course we stack four of them on elated-minsky | 09:57:44 |
| hexa (signing key rotation when) | I'd say we kill them all and I send a fix to master | 10:03:54 |
 K900 | 
Download OK the highlighted output on NixOS tests is REALLY nice | 10:08:14 |
| hexa (signing key rotation when) | that looks much better | 10:08:17 |
| hexa (signing key rotation when) | https://github.com/NixOS/nixpkgs/pull/535575 | 10:24:29 |
