| 27 Jun 2026 |
 hexa | I think missing runnables is really in the queue-runner | 09:23:35 |
| hexa | especially with so many steps it has ingested | 09:26:36 |
|  Defelo joined the room. | 10:46:59 |
 Mic92 | hexa (signing key rotation when): https://github.com/NixOS/infra/pull/1099 My recommendation is to first deploy this on staging, eval nixos unstable small and if logs are clear, we can promote this to prod. I have rebased it now | 11:14:29 |
| Mic92 | Once this is in I can run perf again and see what the new bottleneck is. | 11:14:49 |
| hexa | applying to staging-hydra | 11:15:35 |
| Mic92 | I don't think the current bottlenecks will be very visible in prometheus. But they are very obvious if you open htop | 11:16:50 |
 Grimmauld (any/all) | Can't really open htop without being infra team though. And prometheus does show some info worth looking at imo, even if it doesn't show the reasons behind metrics. | 11:18:16 |
| Mic92 | I was aware of the performance bottleneck when I switched to this code but there were more pressing issues like fixing nix gc and other bugs. | 11:20:10 |
| Mic92 | long-term it would be nice to make the evaluation deamon-less but there is some handover between queue-runner and hydra-eval-job set that needs to happen. Because they run as different user. | 11:21:18 |
| Mic92 | * long-term it would be nice to make the evaluation nix-deamon-less but there is some handover between queue-runner and hydra-eval-job set that needs to happen. Because they run as different user. | 11:21:30 |
| hexa | eval is running https://staging-hydra.nixos.org/jobset/nixos/unstable-small | 11:29:00 |
 eyJhb | @[hexa (signing key rotation when)] how big of a hassle would it be to change the signing keys? | 11:30:42 |
| hexa | not super big | 11:31:04 |
| hexa | requires some testing | 11:31:11 |
| hexa | rolling a new key | 11:31:14 |
| hexa | ideally the new key is pq safe | 11:31:25 |
| hexa | make nixpkgs adopt the new key in addition to the old key | 11:31:51 |
| hexa | set a retire date for the old key, or don't | 11:32:03 |
| hexa | but at some point it should be removed from default trust in nixpkgs/nixos | 11:33:18 |
| hexa | Redacted or Malformed Event | 11:33:21 |
| hexa | we can enumerate the number of people who could've pulled the old signing key off hydra.nixos.org | 11:34:05 |
| hexa | and it is probably a high single digit or low double digit number | 11:34:33 |
| hexa | still no good way to put it into a secure enclave | 11:34:54 |
| hexa | though I'm sure the foundation would grant funding for such a project | 11:35:08 |
| hexa | it doesn't look like nix is bottlenecking on that branch | 11:36:57 |
| hexa | # curl -s localhost:8080/metrics | grep hydraqueuerunner_machine_type_runnable
# HELP hydraqueuerunner_machine_type_runnable Number of runnable build steps per machine type
# TYPE hydraqueuerunner_machine_type_runnable gauge
hydraqueuerunner_machine_type_runnable{machine_type="aarch64-darwin"} 0
hydraqueuerunner_machine_type_runnable{machine_type="aarch64-linux"} 825
hydraqueuerunner_machine_type_runnable{machine_type="builtin"} 0
hydraqueuerunner_machine_type_runnable{machine_type="i686-linux"} 0
hydraqueuerunner_machine_type_runnable{machine_type="x86_64-darwin"} 0
hydraqueuerunner_machine_type_runnable{machine_type="x86_64-linux"} 876
hydraqueuerunner_machine_type_runnable{machine_type="x86_64-v1-linux"} 0
hydraqueuerunner_machine_type_runnable{machine_type="x86_64-v2-linux"} 0
hydraqueuerunner_machine_type_runnable{machine_type="x86_64-v3-linux"} 0
| 11:38:33 |
| hexa | Redacted or Malformed Event | 11:38:45 |
| hexa | that looks sensible | 11:38:48 |
| eyJhb | That was also my theory, thanks for explaining it :) Would be nice if they were rotated, but I wonder how much hassle it would cause some weird edge-case people made. But worst cause, I assume they could just have all the old signing keys :) | 11:42:55 |
