| 17 Jun 2026 |
 leona | and also shop up in the 'running builds' view | 06:10:57 |
 Vladimír Čunát | Those paths are on cache.nixos.org already, at least. | 06:38:06 |
| leona | * and also show up in the 'running builds' view | 06:38:25 |
 hexa | https://paste.lossy.network/raw/CN3FRYPEEHF4N2QTDJXA6UDATY
https://hydra.nixos.org/build/331656644 | 07:53:30 |
| hexa | https://paste.lossy.network/GS3TZVHGHUQGTJET6M7ZP56FFA and from the queue-builder | 07:58:06 |
| hexa | Redacted or Malformed Event | 07:58:18 |
 Mic92 | We need to work on our logging serializer a bit. It has too many hex strings | 08:01:15 |
| hexa | not super clear how to unstuck https://hydra.nixos.org/build/331656644 | 08:23:12 |
| hexa | did a few changes to make the new metrics appear | 08:34:48 |
| hexa | ok, got all graphs hooked up | 08:45:58 |
| hexa | 
Download | 09:25:04 |
| hexa | the latter two seem very stuck | 09:25:07 |
| hexa | I stopped the queue-builder on elated-minsky and they're not going away | 09:25:17 |
| hexa | restarting the queue-runner next | 09:26:11 |
| hexa | ok, that did clear those | 09:26:45 |
| Vladimír Čunát | "Unsupported system type" here is very fishy:
https://hydra.nixos.org/build/331649953#tabs-buildsteps | 10:05:11 |
| Vladimír Čunát | There are weird inconsistencies. Some evals restart jobs which already succeeded. And now e.g. page https://hydra.nixos.org/jobset/nixos/release-26.05 shows cancellations only in the last eval but if click older evals, the cancellations also get shown in there. | 13:02:47 |
| Vladimír Čunát | * There are weird inconsistencies. Some evals restart jobs which already succeeded. And now e.g. page https://hydra.nixos.org/jobset/nixos/release-26.05 shows cancellations only in the last eval but if you click older evals, the cancellations also get shown in there. | 13:03:03 |
 Sergei Zimmerman (xokdvium) | In reply to @vcunat:matrix.org
"Unsupported system type" here is very fishy:
https://hydra.nixos.org/build/331649953#tabs-buildsteps A lot of nix jobs fail similarly:
https://hydra.nixos.org/build/331656015 | 13:22:46 |
| Vladimír Čunát | Oh, could it be meaning i686-linux? | 13:24:11 |
| Sergei Zimmerman (xokdvium) | No, all systems seem to sporadically show this it seems | 13:24:47 |
| hexa | we noticed unsupported system last night, too | 13:36:25 |
| hexa | and I thought it didn't respect max_unsupported_time and aborted builds when builders were unavailable for a short time | 13:36:57 |
| hexa | https://github.com/NixOS/hydra/issues/1795 | 13:36:58 |
| hexa | that may be wrong after all | 13:37:02 |
| hexa | Jun 17 13:41:10 mimas hydra-queue-runner[3159870]: 2026-06-17T13:41:10.510774Z INFO request:complete_build:fail_step_by_uuid:fail_step:finish_build_step: hydra_queue_runner::utils: Writing buildstep result in db. step_status=Aborted start_time=Ok(1781703670) stop_time=Ok(1781703670) method=POST uri=/runner.v1.RunnerService/CompleteBuild version=HTTP/2.0 machine_id="94004b5d-5868-44e5-a9ea-0176b2f15310" build_id="8a61155a-269d-48a3-86fb-005a258b2ce3" state=Completed(UploadFailure) timings=BuildTimings { import_elapsed: 85ms, build_elapsed: 2ms, upload_elapsed: 0ns } machine_id=94004b5d-5868-44e5-a9ea-0176b2f15310 build_id=8a61155a-269d-48a3-86fb-005a258b2ce3 state=Completed(UploadFailure) timings=BuildTimings { import_elapsed: 85ms, build_elapsed: 2ms, upload_elapsed: 0ns } machine_id=94004b5d-5868-44e5-a9ea-0176b2f15310 drv_path=15w46vg2clddz8k52dkxq4m3vrrz4gfz-httptoolkit-server-1.26.1.drv build_id=331168100 step_nr=6 machine=Some("elated-minsky") output_paths=None
| 13:42:08 |
| hexa |
step_status=Aborted
state=Completed(UploadFailure)
| 13:42:36 |
| Sergei Zimmerman (xokdvium) | S3 failures again? | 13:44:44 |
| hexa | I wish it would be clearer about HTTP errors here, but that feels very likely | 13:45:23 |
| hexa | and now we need to understand if it evn retries uploads | 13:45:32 |
