| 17 Jun 2026 |
 hexa (signing key rotation when) | I wish it would be clearer about HTTP errors here, but that feels very likely | 13:45:23 |
| hexa (signing key rotation when) | and now we need to understand if it evn retries uploads | 13:45:32 |
 Sergei Zimmerman (xokdvium) | I looked at the object_store crate a while back and it seemed like it has zero retry logic for issues that plagued us? | 13:46:43 |
| hexa (signing key rotation when) | https://paste.lossy.network/raw/32U3OGAE3PEZRCQW2KNBIFQXXQ the full queue-runner log for that deriv | 13:47:16 |
 K900 | Can you get the logs from the machine actually doing the uploading? | 13:47:51 |
| K900 | ...wait do we just | 13:47:52 |
| K900 | Not have centralized logging | 13:47:55 |
| K900 | From the builders | 13:47:57 |
| hexa (signing key rotation when) | https://paste.lossy.network/raw/DHAYKQFKLGUTYGTOPB4FC5DYIA from the queue-builder | 13:48:49 |
| hexa (signing key rotation when) | oops, that is cut off | 13:49:16 |
| K900 | Jun 17 13:44:10 elated-minsky hydra-builder[973519]: 2026-06-17T13:44:10.950607Z ERROR process_build:upload_nars: hydra_builder::state: error=QueryPathInfo: io error No discriminant in enum RawLogMessageType matches the value 8031170683229269551 timings=BuildTimings { import_elapsed: 0ns, build_elapsed: 0ns, upload_elapsed: 0ns } drv=Some(ProtoStorePath(StorePath(15w46vg2clddz8k52dkxq4m3vrrz4gfz-httptoolkit-server-1.26.1.drv))) build_id="f3b4bedc-9b86-4f55-89e5-c3bf11cd3f09" machine_id="94004b5d-5868-44e5-a9ea-0176b2f15310" presigned_url_opts=Some(PresignedUploadOpts { upload_debug_info: false })
| 13:51:05 |
| K900 | This looks mighty sus | 13:51:15 |
| K900 | >>> binascii.unhexlify(hex(8031170683229269551)[2:])
b'ots/xin/'
| 13:51:24 |
| K900 | Yeah that's turbo sus | 13:51:31 |
| hexa (signing key rotation when) | 2026-06-17T13:44:10.951230Z ERROR hydra_builder::state: Build of 15w46vg2clddz8k52dkxq4m3vrrz4gfz-httptoolkit-server-1.26.1.drv failed with:
0: \x1b[91mUpload failure\x1b[0m
1: \x1b[91mQueryPathInfo: io error No discriminant in enum `RawLogMessageType` matches the value `8031170683229269551`\x1b[0m
2: \x1b[91mNo discriminant in enum `RawLogMessageType` matches the value `8031170683229269551`\x1b[0m
Location:
\x1b[35msubprojects/hydra-builder/src/state.rs\x1b[0m:\x1b[35m330\x1b[0m
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
\x1b[96m ⋮ 3 frames hidden ⋮ \x1b[0m
4: \x1b[91mhydra_builder::state::State::schedule_build::{{closure}}\x1b[0m\x1b[90m::he33d915f497cbb27\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
5: \x1b[91mtokio::runtime::task::core::Core<T,S>::poll\x1b[0m\x1b[90m::h004ab324c77f4d41\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
6: \x1b[91mtokio::runtime::task::harness::Harness<T,S>::poll\x1b[0m\x1b[90m::h04b93015b79529a7\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
7: \x1b[91mtokio::runtime::scheduler::multi_thread::worker::Context::run_task\x1b[0m\x1b[90m::h3db64955b150046b\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
8: \x1b[91mtokio::runtime::context::scoped::Scoped<T>::set\x1b[0m\x1b[90m::h28df0b8f59cd44a7\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
9: \x1b[91mtokio::runtime::context::runtime::enter_runtime\x1b[0m\x1b[90m::h09abbc49fd9942be\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
10: \x1b[91mtokio::runtime::scheduler::multi_thread::worker::run\x1b[0m\x1b[90m::hc1c927e546434dc4\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
11: \x1b[91m<tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll\x1b[0m\x1b[90m::h7f3b19029431281c\x1b[
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
12: \x1b[91mtokio::runtime::task::core::Core<T,S>::poll\x1b[0m\x1b[90m::h6121ec923869a09d\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
13: \x1b[91mtokio::runtime::task::harness::Harness<T,S>::poll\x1b[0m\x1b[90m::hac42bd960269d040\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
14: \x1b[91mtokio::runtime::blocking::pool::Inner::run\x1b[0m\x1b[90m::haa7e4d8d38d20e2c\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
15: \x1b[32mstd::sys::backtrace::__rust_begin_short_backtrace\x1b[0m\x1b[90m::h0f806bbae0347e6e\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
16: \x1b[32mcore::ops::function::FnOnce::call_once{{vtable.shim}}\x1b[0m\x1b[90m::ha4f94bf82ab74a77\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
17: \x1b[91m<std[e114a732dcca3553]::sys::thread::unix::Thread>::new::thread_start\x1b[0m\x1b[90m<unknown>\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
18: \x1b[32mstart_thread\x1b[0m\x1b[90m<unknown>\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
19: \x1b[91m__GI___clone\x1b[0m\x1b[90m<unknown>\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.
Run with RUST_BACKTRACE=full to include source snippets.
| 13:51:54 |
| K900 | It's definitely reading /nix/store | 13:52:35 |
| K900 | Instead of whatever it's supposed to be readin | 13:52:37 |
| K900 | Instead of whatever it's supposed to be reading | 13:52:38 |
| K900 | Well, /nix/sto | 13:52:44 |
| K900 | Technically we can't prove what comes next | 13:52:48 |
| K900 | But I am fairly certain | 13:52:57 |
| hexa (signing key rotation when) | can someone remind me, how was the old queue-runner. did it complete builds and whether or not the upload failed did not affect the job state? | 14:10:01 |
| hexa (signing key rotation when) | and maybe we're now getting much more visibility into upload problems, because upload problems abort builds. | 14:10:29 |
| K900 | We had aborted builds due to upload failures before | 14:11:28 |
| K900 | I'm pretty sure this isn't a real upload failure | 14:11:42 |
| K900 | But whatever that parse error is | 14:11:42 |
| hexa (signing key rotation when) | yeah, I'd like a more convincing error message than
No discriminant in enum RawLogMessageType matches the value 8031170683229269551
| 14:13:32 |
| K900 | ...I have a theory actually | 14:14:30 |
| K900 | What are the odds it's trying to parse the log protocol | 14:14:45 |
| K900 | And instead it just gets nothing and then a store path | 14:14:50 |
