| 13 Apr 2025 |
 hexa | Apr 13 14:21:00 mimas hydra-evaluator[4157649]: (nixpkgs:staging) Fetching input `nixpkgs` (git) https://github.com/nixos/nixpkgs.git staging-24.11
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: (nixpkgs:staging) Fetching input `officialRelease` (boolean) false
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: (nixpkgs:staging) Evaluating...
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: warning: unknown setting 'allowed-users'
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: warning: unknown setting 'trusted-users'
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: error: file 'nixpkgs/pkgs/top-level/release-staging.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: error: file 'nixpkgs/pkgs/top-level/release-staging.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: error: file 'nixpkgs/pkgs/top-level/release-staging.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: error: file 'nixpkgs/pkgs/top-level/release-staging.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: error: file 'nixpkgs/pkgs/top-level/release-staging.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: error: file 'nixpkgs/pkgs/top-level/release-staging.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: error: file 'nixpkgs/pkgs/top-level/release-staging.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: error: file 'nixpkgs/pkgs/top-level/release-staging.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: error: file 'nixpkgs/pkgs/top-level/release-staging.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: error: file 'nixpkgs/pkgs/top-level/release-staging.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: error: file 'nixpkgs/pkgs/top-level/release-staging.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: error: file 'nixpkgs/pkgs/top-level/release-staging.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: error: file 'nixpkgs/pkgs/top-level/release-staging.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: error: file 'nixpkgs/pkgs/top-level/release-staging.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: error: file 'nixpkgs/pkgs/top-level/release-staging.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: error: file 'nixpkgs/pkgs/top-level/release-staging.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: error: worker error: error: file 'nixpkgs/pkgs/top-level/release-staging.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)
Apr 13 14:21:01 mimas hydra-evaluator[4157649]: process ended prematurely at /nix/store/jsz0p636jy1fxnbci7rra0w6kn8ilysp-hydra-0.1.20250411.f54fa72/bin/.hydra-eval-jobset-wrapped line 405.
Apr 13 14:21:01 mimas hydra-evaluator[3599633]: evaluation of jobset ‘nixpkgs:staging (jobset#125)’ failed with exit code 1
| 14:22:54 |
| hexa | is that a bad clone or what happend? | 14:23:15 |
 Vladimír Čunát | I believe it is quite new on hydra.nixos.org but maybe we've skipped some versions that your instance did not. | 14:23:34 |
| hexa | meh. | 14:23:55 |
| hexa | I had been tracking the same branch as on h.n.o privately, so it is probably something that's been in there for at least a year 😄 | 14:24:37 |
| Vladimír Čunát | I'm watching that page all the time. | 14:25:17 |
| Vladimír Čunát | There's no way I would be missing it for a whole year. | 14:25:27 |
| hexa | yeah, I have rarely seen it happen on h.n.o | 14:25:31 |
| hexa | but iirc it has happend before, but rarely | 14:25:39 |
| hexa | for my private hydra … pretty regularly | 14:25:46 |
| Vladimír Čunát | Maybe some state is involved... | 14:25:47 |
| Vladimír Čunát | * Maybe some state or config is involved... | 14:26:03 |
| hexa | https://github.com/NixOS/hydra/pull/1475 | 14:26:17 |
| hexa | all other patches got picked, just not this one apparently | 14:26:44 |
| hexa | Redacted or Malformed Event | 14:26:47 |
| hexa | hm no, the metrics are there | 14:29:01 |
| hexa | mergiraf-- | 14:29:04 |
| hexa | https://github.com/NixOS/hydra/blob/master/src/hydra-queue-runner/state.hh#L461-L465 | 14:30:07 |
| hexa | https://github.com/NixOS/hydra/blob/master/src/hydra-queue-runner/hydra-queue-runner.cc#L73-L100 | 14:30:23 |
| hexa | $ curl http://hydra.nixos.org:9198/metrics
# HELP exposer_transferred_bytes_total Transferred bytes to metrics services
# TYPE exposer_transferred_bytes_total counter
exposer_transferred_bytes_total 6181291
# HELP exposer_scrapes_total Number of times metrics were scraped
# TYPE exposer_scrapes_total counter
exposer_scrapes_total 12227
# HELP exposer_request_latencies Latencies of serving scrape requests, in microseconds
# TYPE exposer_request_latencies summary
exposer_request_latencies_count 12227
exposer_request_latencies_sum 41993705
exposer_request_latencies{quantile="0.5"} 321
exposer_request_latencies{quantile="0.9"} 321
exposer_request_latencies{quantile="0.99"} 321
# HELP hydraqueuerunner_queue_checks_started_total Number of times State::getQueuedBuilds() was started
# TYPE hydraqueuerunner_queue_checks_started_total counter
hydraqueuerunner_queue_checks_started_total 12821
# HELP hydraqueuerunner_queue_build_loads_total Number of builds loaded
# TYPE hydraqueuerunner_queue_build_loads_total counter
hydraqueuerunner_queue_build_loads_total 75140
# HELP hydraqueuerunner_queue_steps_created_total Number of steps created
# TYPE hydraqueuerunner_queue_steps_created_total counter
hydraqueuerunner_queue_steps_created_total 539634
# HELP hydraqueuerunner_queue_checks_early_exits_total Number of times State::getQueuedBuilds() yielded to potential bumps
# TYPE hydraqueuerunner_queue_checks_early_exits_total counter
hydraqueuerunner_queue_checks_early_exits_total 81
# HELP hydraqueuerunner_queue_checks_finished_total Number of times State::getQueuedBuilds() was completed
# TYPE hydraqueuerunner_queue_checks_finished_total counter
hydraqueuerunner_queue_checks_finished_total 12820
# HELP hydraqueuerunner_queue_max_build_id_info Maximum build record ID in the queue
# TYPE hydraqueuerunner_queue_max_build_id_info gauge
hydraqueuerunner_queue_max_build_id_info 294707506
| 14:31:52 |
| hexa | the dispatcher hooks are also still here https://github.com/NixOS/hydra/blob/master/src/hydra-queue-runner/dispatcher.cc#L49-L67 | 14:35:04 |
| hexa | so are the queue monitor hooks https://github.com/NixOS/hydra/blob/master/src/hydra-queue-runner/queue-monitor.cc#L54-L82 | 14:35:48 |
| hexa | the bottleneck metrics went missing on 2025-04-11 around 10:30 UTC | 14:43:58 |
| hexa | but the last hydra bump was already on 2025-04-08 | 14:44:22 |
| hexa | ● hydra-queue-runner.service
Loaded: loaded (/etc/systemd/system/hydra-queue-runner.service; enabled; preset: ignored)
Active: active (running) since Fri 2025-04-11 11:34:56 UTC; 2 days ago
| 14:44:53 |
| hexa | * the bottleneck metrics went missing on 2025-04-11 around 11:30 UTC | 14:44:57 |
| hexa | so this broke with a queue-runner restart? | 14:45:09 |
| Vladimír Čunát | If the runner didn't restart, it's been running on the old version until then. | 14:46:17 |
| hexa | makes sense | 14:47:18 |
| Vladimír Čunát | IIRC it has an exception not to restart by default. | 14:48:10 |
