| 13 Apr 2025 |
 hexa (signing key rotation when) | yeah, I have rarely seen it happen on h.n.o | 14:25:31 |
| hexa (signing key rotation when) | but iirc it has happend before, but rarely | 14:25:39 |
| hexa (signing key rotation when) | for my private hydra … pretty regularly | 14:25:46 |
 vcunat | Maybe some state is involved... | 14:25:47 |
| vcunat | * Maybe some state or config is involved... | 14:26:03 |
| hexa (signing key rotation when) | https://github.com/NixOS/hydra/pull/1475 | 14:26:17 |
| hexa (signing key rotation when) | all other patches got picked, just not this one apparently | 14:26:44 |
| hexa (signing key rotation when) | Redacted or Malformed Event | 14:26:47 |
| hexa (signing key rotation when) | hm no, the metrics are there | 14:29:01 |
| hexa (signing key rotation when) | mergiraf-- | 14:29:04 |
| hexa (signing key rotation when) | https://github.com/NixOS/hydra/blob/master/src/hydra-queue-runner/state.hh#L461-L465 | 14:30:07 |
| hexa (signing key rotation when) | https://github.com/NixOS/hydra/blob/master/src/hydra-queue-runner/hydra-queue-runner.cc#L73-L100 | 14:30:23 |
| hexa (signing key rotation when) | $ curl http://hydra.nixos.org:9198/metrics
# HELP exposer_transferred_bytes_total Transferred bytes to metrics services
# TYPE exposer_transferred_bytes_total counter
exposer_transferred_bytes_total 6181291
# HELP exposer_scrapes_total Number of times metrics were scraped
# TYPE exposer_scrapes_total counter
exposer_scrapes_total 12227
# HELP exposer_request_latencies Latencies of serving scrape requests, in microseconds
# TYPE exposer_request_latencies summary
exposer_request_latencies_count 12227
exposer_request_latencies_sum 41993705
exposer_request_latencies{quantile="0.5"} 321
exposer_request_latencies{quantile="0.9"} 321
exposer_request_latencies{quantile="0.99"} 321
# HELP hydraqueuerunner_queue_checks_started_total Number of times State::getQueuedBuilds() was started
# TYPE hydraqueuerunner_queue_checks_started_total counter
hydraqueuerunner_queue_checks_started_total 12821
# HELP hydraqueuerunner_queue_build_loads_total Number of builds loaded
# TYPE hydraqueuerunner_queue_build_loads_total counter
hydraqueuerunner_queue_build_loads_total 75140
# HELP hydraqueuerunner_queue_steps_created_total Number of steps created
# TYPE hydraqueuerunner_queue_steps_created_total counter
hydraqueuerunner_queue_steps_created_total 539634
# HELP hydraqueuerunner_queue_checks_early_exits_total Number of times State::getQueuedBuilds() yielded to potential bumps
# TYPE hydraqueuerunner_queue_checks_early_exits_total counter
hydraqueuerunner_queue_checks_early_exits_total 81
# HELP hydraqueuerunner_queue_checks_finished_total Number of times State::getQueuedBuilds() was completed
# TYPE hydraqueuerunner_queue_checks_finished_total counter
hydraqueuerunner_queue_checks_finished_total 12820
# HELP hydraqueuerunner_queue_max_build_id_info Maximum build record ID in the queue
# TYPE hydraqueuerunner_queue_max_build_id_info gauge
hydraqueuerunner_queue_max_build_id_info 294707506
| 14:31:52 |
| hexa (signing key rotation when) | the dispatcher hooks are also still here https://github.com/NixOS/hydra/blob/master/src/hydra-queue-runner/dispatcher.cc#L49-L67 | 14:35:04 |
| hexa (signing key rotation when) | so are the queue monitor hooks https://github.com/NixOS/hydra/blob/master/src/hydra-queue-runner/queue-monitor.cc#L54-L82 | 14:35:48 |
| hexa (signing key rotation when) | the bottleneck metrics went missing on 2025-04-11 around 10:30 UTC | 14:43:58 |
| hexa (signing key rotation when) | but the last hydra bump was already on 2025-04-08 | 14:44:22 |
| hexa (signing key rotation when) | ● hydra-queue-runner.service
Loaded: loaded (/etc/systemd/system/hydra-queue-runner.service; enabled; preset: ignored)
Active: active (running) since Fri 2025-04-11 11:34:56 UTC; 2 days ago
| 14:44:53 |
| hexa (signing key rotation when) | * the bottleneck metrics went missing on 2025-04-11 around 11:30 UTC | 14:44:57 |
| hexa (signing key rotation when) | so this broke with a queue-runner restart? | 14:45:09 |
| vcunat | If the runner didn't restart, it's been running on the old version until then. | 14:46:17 |
| hexa (signing key rotation when) | makes sense | 14:47:18 |
| vcunat | IIRC it has an exception not to restart by default. | 14:48:10 |
| vcunat | (as it's relatively expensive) | 14:48:18 |
| hexa (signing key rotation when) | yes | 14:48:21 |
| vcunat | * (as it can be relatively expensive) | 14:48:25 |
| hexa (signing key rotation when) | reported here https://github.com/NixOS/hydra/issues/1476 | 14:57:07 |
| hexa (signing key rotation when) | and https://github.com/NixOS/hydra/issues/1477 | 15:03:06 |
| hexa (signing key rotation when) | Redacted or Malformed Event | 19:16:57 |
| hexa (signing key rotation when) | Redacted or Malformed Event | 19:17:01 |
