| 13 Apr 2025 |
 hexa | Redacted or Malformed Event | 14:26:47 |
| hexa | hm no, the metrics are there | 14:29:01 |
| hexa | mergiraf-- | 14:29:04 |
| hexa | https://github.com/NixOS/hydra/blob/master/src/hydra-queue-runner/state.hh#L461-L465 | 14:30:07 |
| hexa | https://github.com/NixOS/hydra/blob/master/src/hydra-queue-runner/hydra-queue-runner.cc#L73-L100 | 14:30:23 |
| hexa | $ curl http://hydra.nixos.org:9198/metrics
# HELP exposer_transferred_bytes_total Transferred bytes to metrics services
# TYPE exposer_transferred_bytes_total counter
exposer_transferred_bytes_total 6181291
# HELP exposer_scrapes_total Number of times metrics were scraped
# TYPE exposer_scrapes_total counter
exposer_scrapes_total 12227
# HELP exposer_request_latencies Latencies of serving scrape requests, in microseconds
# TYPE exposer_request_latencies summary
exposer_request_latencies_count 12227
exposer_request_latencies_sum 41993705
exposer_request_latencies{quantile="0.5"} 321
exposer_request_latencies{quantile="0.9"} 321
exposer_request_latencies{quantile="0.99"} 321
# HELP hydraqueuerunner_queue_checks_started_total Number of times State::getQueuedBuilds() was started
# TYPE hydraqueuerunner_queue_checks_started_total counter
hydraqueuerunner_queue_checks_started_total 12821
# HELP hydraqueuerunner_queue_build_loads_total Number of builds loaded
# TYPE hydraqueuerunner_queue_build_loads_total counter
hydraqueuerunner_queue_build_loads_total 75140
# HELP hydraqueuerunner_queue_steps_created_total Number of steps created
# TYPE hydraqueuerunner_queue_steps_created_total counter
hydraqueuerunner_queue_steps_created_total 539634
# HELP hydraqueuerunner_queue_checks_early_exits_total Number of times State::getQueuedBuilds() yielded to potential bumps
# TYPE hydraqueuerunner_queue_checks_early_exits_total counter
hydraqueuerunner_queue_checks_early_exits_total 81
# HELP hydraqueuerunner_queue_checks_finished_total Number of times State::getQueuedBuilds() was completed
# TYPE hydraqueuerunner_queue_checks_finished_total counter
hydraqueuerunner_queue_checks_finished_total 12820
# HELP hydraqueuerunner_queue_max_build_id_info Maximum build record ID in the queue
# TYPE hydraqueuerunner_queue_max_build_id_info gauge
hydraqueuerunner_queue_max_build_id_info 294707506
| 14:31:52 |
| hexa | the dispatcher hooks are also still here https://github.com/NixOS/hydra/blob/master/src/hydra-queue-runner/dispatcher.cc#L49-L67 | 14:35:04 |
| hexa | so are the queue monitor hooks https://github.com/NixOS/hydra/blob/master/src/hydra-queue-runner/queue-monitor.cc#L54-L82 | 14:35:48 |
| hexa | the bottleneck metrics went missing on 2025-04-11 around 10:30 UTC | 14:43:58 |
| hexa | but the last hydra bump was already on 2025-04-08 | 14:44:22 |
