| 17 Jun 2026 |
 hexa | restarting the queue-runner next | 09:26:11 |
| hexa | ok, that did clear those | 09:26:45 |
 Vladimír Čunát | "Unsupported system type" here is very fishy:
https://hydra.nixos.org/build/331649953#tabs-buildsteps | 10:05:11 |
| Vladimír Čunát | There are weird inconsistencies. Some evals restart jobs which already succeeded. And now e.g. page https://hydra.nixos.org/jobset/nixos/release-26.05 shows cancellations only in the last eval but if click older evals, the cancellations also get shown in there. | 13:02:47 |
| Vladimír Čunát | * There are weird inconsistencies. Some evals restart jobs which already succeeded. And now e.g. page https://hydra.nixos.org/jobset/nixos/release-26.05 shows cancellations only in the last eval but if you click older evals, the cancellations also get shown in there. | 13:03:03 |
 Sergei Zimmerman (xokdvium) | In reply to @vcunat:matrix.org
"Unsupported system type" here is very fishy:
https://hydra.nixos.org/build/331649953#tabs-buildsteps A lot of nix jobs fail similarly:
https://hydra.nixos.org/build/331656015 | 13:22:46 |
| Vladimír Čunát | Oh, could it be meaning i686-linux? | 13:24:11 |
| Sergei Zimmerman (xokdvium) | No, all systems seem to sporadically show this it seems | 13:24:47 |
| hexa | we noticed unsupported system last night, too | 13:36:25 |
| hexa | and I thought it didn't respect max_unsupported_time and aborted builds when builders were unavailable for a short time | 13:36:57 |
| hexa | https://github.com/NixOS/hydra/issues/1795 | 13:36:58 |
| hexa | that may be wrong after all | 13:37:02 |
| hexa | Jun 17 13:41:10 mimas hydra-queue-runner[3159870]: 2026-06-17T13:41:10.510774Z INFO request:complete_build:fail_step_by_uuid:fail_step:finish_build_step: hydra_queue_runner::utils: Writing buildstep result in db. step_status=Aborted start_time=Ok(1781703670) stop_time=Ok(1781703670) method=POST uri=/runner.v1.RunnerService/CompleteBuild version=HTTP/2.0 machine_id="94004b5d-5868-44e5-a9ea-0176b2f15310" build_id="8a61155a-269d-48a3-86fb-005a258b2ce3" state=Completed(UploadFailure) timings=BuildTimings { import_elapsed: 85ms, build_elapsed: 2ms, upload_elapsed: 0ns } machine_id=94004b5d-5868-44e5-a9ea-0176b2f15310 build_id=8a61155a-269d-48a3-86fb-005a258b2ce3 state=Completed(UploadFailure) timings=BuildTimings { import_elapsed: 85ms, build_elapsed: 2ms, upload_elapsed: 0ns } machine_id=94004b5d-5868-44e5-a9ea-0176b2f15310 drv_path=15w46vg2clddz8k52dkxq4m3vrrz4gfz-httptoolkit-server-1.26.1.drv build_id=331168100 step_nr=6 machine=Some("elated-minsky") output_paths=None
| 13:42:08 |
| hexa |
step_status=Aborted
state=Completed(UploadFailure)
| 13:42:36 |
| Sergei Zimmerman (xokdvium) | S3 failures again? | 13:44:44 |
| hexa | I wish it would be clearer about HTTP errors here, but that feels very likely | 13:45:23 |
| hexa | and now we need to understand if it evn retries uploads | 13:45:32 |
| Sergei Zimmerman (xokdvium) | I looked at the object_store crate a while back and it seemed like it has zero retry logic for issues that plagued us? | 13:46:43 |
| hexa | https://paste.lossy.network/raw/32U3OGAE3PEZRCQW2KNBIFQXXQ the full queue-runner log for that deriv | 13:47:16 |
 K900 | Can you get the logs from the machine actually doing the uploading? | 13:47:51 |
| K900 | ...wait do we just | 13:47:52 |
| K900 | Not have centralized logging | 13:47:55 |
| K900 | From the builders | 13:47:57 |
| hexa | https://paste.lossy.network/raw/DHAYKQFKLGUTYGTOPB4FC5DYIA from the queue-builder | 13:48:49 |
| hexa | oops, that is cut off | 13:49:16 |
| K900 | Jun 17 13:44:10 elated-minsky hydra-builder[973519]: 2026-06-17T13:44:10.950607Z ERROR process_build:upload_nars: hydra_builder::state: error=QueryPathInfo: io error No discriminant in enum RawLogMessageType matches the value 8031170683229269551 timings=BuildTimings { import_elapsed: 0ns, build_elapsed: 0ns, upload_elapsed: 0ns } drv=Some(ProtoStorePath(StorePath(15w46vg2clddz8k52dkxq4m3vrrz4gfz-httptoolkit-server-1.26.1.drv))) build_id="f3b4bedc-9b86-4f55-89e5-c3bf11cd3f09" machine_id="94004b5d-5868-44e5-a9ea-0176b2f15310" presigned_url_opts=Some(PresignedUploadOpts { upload_debug_info: false })
| 13:51:05 |
| K900 | This looks mighty sus | 13:51:15 |
| K900 | >>> binascii.unhexlify(hex(8031170683229269551)[2:])
b'ots/xin/'
| 13:51:24 |
| K900 | Yeah that's turbo sus | 13:51:31 |
| hexa | 2026-06-17T13:44:10.951230Z ERROR hydra_builder::state: Build of 15w46vg2clddz8k52dkxq4m3vrrz4gfz-httptoolkit-server-1.26.1.drv failed with:
0: \x1b[91mUpload failure\x1b[0m
1: \x1b[91mQueryPathInfo: io error No discriminant in enum `RawLogMessageType` matches the value `8031170683229269551`\x1b[0m
2: \x1b[91mNo discriminant in enum `RawLogMessageType` matches the value `8031170683229269551`\x1b[0m
Location:
\x1b[35msubprojects/hydra-builder/src/state.rs\x1b[0m:\x1b[35m330\x1b[0m
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
\x1b[96m ⋮ 3 frames hidden ⋮ \x1b[0m
4: \x1b[91mhydra_builder::state::State::schedule_build::{{closure}}\x1b[0m\x1b[90m::he33d915f497cbb27\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
5: \x1b[91mtokio::runtime::task::core::Core<T,S>::poll\x1b[0m\x1b[90m::h004ab324c77f4d41\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
6: \x1b[91mtokio::runtime::task::harness::Harness<T,S>::poll\x1b[0m\x1b[90m::h04b93015b79529a7\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
7: \x1b[91mtokio::runtime::scheduler::multi_thread::worker::Context::run_task\x1b[0m\x1b[90m::h3db64955b150046b\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
8: \x1b[91mtokio::runtime::context::scoped::Scoped<T>::set\x1b[0m\x1b[90m::h28df0b8f59cd44a7\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
9: \x1b[91mtokio::runtime::context::runtime::enter_runtime\x1b[0m\x1b[90m::h09abbc49fd9942be\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
10: \x1b[91mtokio::runtime::scheduler::multi_thread::worker::run\x1b[0m\x1b[90m::hc1c927e546434dc4\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
11: \x1b[91m<tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll\x1b[0m\x1b[90m::h7f3b19029431281c\x1b[
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
12: \x1b[91mtokio::runtime::task::core::Core<T,S>::poll\x1b[0m\x1b[90m::h6121ec923869a09d\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
13: \x1b[91mtokio::runtime::task::harness::Harness<T,S>::poll\x1b[0m\x1b[90m::hac42bd960269d040\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
14: \x1b[91mtokio::runtime::blocking::pool::Inner::run\x1b[0m\x1b[90m::haa7e4d8d38d20e2c\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
15: \x1b[32mstd::sys::backtrace::__rust_begin_short_backtrace\x1b[0m\x1b[90m::h0f806bbae0347e6e\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
16: \x1b[32mcore::ops::function::FnOnce::call_once{{vtable.shim}}\x1b[0m\x1b[90m::ha4f94bf82ab74a77\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
17: \x1b[91m<std[e114a732dcca3553]::sys::thread::unix::Thread>::new::thread_start\x1b[0m\x1b[90m<unknown>\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
18: \x1b[32mstart_thread\x1b[0m\x1b[90m<unknown>\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
19: \x1b[91m__GI___clone\x1b[0m\x1b[90m<unknown>\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.
Run with RUST_BACKTRACE=full to include source snippets.
| 13:51:54 |
