| 17 Jun 2026 |
 K900 | >>> binascii.unhexlify(hex(8031170683229269551)[2:])
b'ots/xin/'
| 13:51:24 |
| K900 | Yeah that's turbo sus | 13:51:31 |
 hexa | 2026-06-17T13:44:10.951230Z ERROR hydra_builder::state: Build of 15w46vg2clddz8k52dkxq4m3vrrz4gfz-httptoolkit-server-1.26.1.drv failed with:
0: \x1b[91mUpload failure\x1b[0m
1: \x1b[91mQueryPathInfo: io error No discriminant in enum `RawLogMessageType` matches the value `8031170683229269551`\x1b[0m
2: \x1b[91mNo discriminant in enum `RawLogMessageType` matches the value `8031170683229269551`\x1b[0m
Location:
\x1b[35msubprojects/hydra-builder/src/state.rs\x1b[0m:\x1b[35m330\x1b[0m
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
\x1b[96m ⋮ 3 frames hidden ⋮ \x1b[0m
4: \x1b[91mhydra_builder::state::State::schedule_build::{{closure}}\x1b[0m\x1b[90m::he33d915f497cbb27\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
5: \x1b[91mtokio::runtime::task::core::Core<T,S>::poll\x1b[0m\x1b[90m::h004ab324c77f4d41\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
6: \x1b[91mtokio::runtime::task::harness::Harness<T,S>::poll\x1b[0m\x1b[90m::h04b93015b79529a7\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
7: \x1b[91mtokio::runtime::scheduler::multi_thread::worker::Context::run_task\x1b[0m\x1b[90m::h3db64955b150046b\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
8: \x1b[91mtokio::runtime::context::scoped::Scoped<T>::set\x1b[0m\x1b[90m::h28df0b8f59cd44a7\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
9: \x1b[91mtokio::runtime::context::runtime::enter_runtime\x1b[0m\x1b[90m::h09abbc49fd9942be\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
10: \x1b[91mtokio::runtime::scheduler::multi_thread::worker::run\x1b[0m\x1b[90m::hc1c927e546434dc4\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
11: \x1b[91m<tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll\x1b[0m\x1b[90m::h7f3b19029431281c\x1b[
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
12: \x1b[91mtokio::runtime::task::core::Core<T,S>::poll\x1b[0m\x1b[90m::h6121ec923869a09d\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
13: \x1b[91mtokio::runtime::task::harness::Harness<T,S>::poll\x1b[0m\x1b[90m::hac42bd960269d040\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
14: \x1b[91mtokio::runtime::blocking::pool::Inner::run\x1b[0m\x1b[90m::haa7e4d8d38d20e2c\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
15: \x1b[32mstd::sys::backtrace::__rust_begin_short_backtrace\x1b[0m\x1b[90m::h0f806bbae0347e6e\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
16: \x1b[32mcore::ops::function::FnOnce::call_once{{vtable.shim}}\x1b[0m\x1b[90m::ha4f94bf82ab74a77\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
17: \x1b[91m<std[e114a732dcca3553]::sys::thread::unix::Thread>::new::thread_start\x1b[0m\x1b[90m<unknown>\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
18: \x1b[32mstart_thread\x1b[0m\x1b[90m<unknown>\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
19: \x1b[91m__GI___clone\x1b[0m\x1b[90m<unknown>\x1b[0m
at \x1b[35m<unknown source file>\x1b[0m:\x1b[35m<unknown line>\x1b[0m
Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.
Run with RUST_BACKTRACE=full to include source snippets.
| 13:51:54 |
| K900 | It's definitely reading /nix/store | 13:52:35 |
| K900 | Instead of whatever it's supposed to be readin | 13:52:37 |
| K900 | Instead of whatever it's supposed to be reading | 13:52:38 |
| K900 | Well, /nix/sto | 13:52:44 |
| K900 | Technically we can't prove what comes next | 13:52:48 |
| K900 | But I am fairly certain | 13:52:57 |
| hexa | can someone remind me, how was the old queue-runner. did it complete builds and whether or not the upload failed did not affect the job state? | 14:10:01 |
| hexa | and maybe we're now getting much more visibility into upload problems, because upload problems abort builds. | 14:10:29 |
| K900 | We had aborted builds due to upload failures before | 14:11:28 |
| K900 | I'm pretty sure this isn't a real upload failure | 14:11:42 |
| K900 | But whatever that parse error is | 14:11:42 |
| hexa | yeah, I'd like a more convincing error message than
No discriminant in enum RawLogMessageType matches the value 8031170683229269551
| 14:13:32 |
| K900 | ...I have a theory actually | 14:14:30 |
| K900 | What are the odds it's trying to parse the log protocol | 14:14:45 |
| K900 | And instead it just gets nothing and then a store path | 14:14:50 |
| K900 | Because the store path already exists | 14:15:00 |
| hexa | https://github.com/NixOS/hydra/blob/f69a55cf3c73146ba4ac4d5e7d39377a226d85fb/subprojects/hydra-builder/src/state.rs#L629-L639 upload_nars called here | 14:19:26 |
| hexa | and I don't see any retry logic | 14:20:09 |
| hexa | not in upload_nars, not in upload_nars_presigned | 14:20:39 |
| hexa | we deployed another update to fix grpc connection aborts from nginx | 16:46:27 |
| K900 | https://hydra.nixos.org/build/331658388 | 18:03:29 |
| K900 | Another unsupported system situation | 18:03:33 |
| hexa | 
Download | 23:22:23 |
| hexa | uhhhhhhh | 23:22:27 |
| hexa | 160k rebuilds seems a bit excessive here | 23:22:58 |
| hexa | ❯ nix-diff /nix/store/m74651b793zgyyvlk9gx7v1cl1ywslib-hello-2.12.3.drv /nix/store/gh9rl6g6jw0kvkzi3mh40ig2phffrwjz-hello-2.12.3.drv
- /nix/store/m74651b793zgyyvlk9gx7v1cl1ywslib-hello-2.12.3.drv:{out}
+ /nix/store/gh9rl6g6jw0kvkzi3mh40ig2phffrwjz-hello-2.12.3.drv:{out}
• The input derivation named `hello-2.12.3.tar.gz` differs
- /nix/store/jk8228a88vjsa0zk004sgdhxp5g4pnnv-hello-2.12.3.tar.gz.drv:{out}
+ /nix/store/b1nwvxqz0prj6kqvwfs7i8d7sk40m4w7-hello-2.12.3.tar.gz.drv:{out}
• The input derivation named `mirrors-list` differs
- /nix/store/84w0zzmqzl2bn5s4rx7axsh9znws1rdf-mirrors-list.drv:{out}
+ /nix/store/qkrfwhcrn7af1w11gxh3m45xjsgxlrxz-mirrors-list.drv:{out}
• The environments do not match:
+ texhistoric=https://ftp.math.utah.edu/pub/tex/historic/ https://texlive.info/historic/ https://ftp.tu-chemnitz.de/pub/tug/historic/ https://pi.kwarc.info/historic/ https://mirrors.tuna.tsinghua.edu.cn/tex-historic-archive/ https://mirror.nju.edu.cn/tex-historic/ ftp://tug.org/texlive/historic/
• Skipping environment comparison
• Skipping environment comparison
| 23:26:37 |
| hexa | mirror list change looks sus | 23:26:41 |
