| 21 Feb 2025 |
 soopyc (she/it) | wack, everything works now for some reason :p sorry for the false alarm; could very well be a temporary blip. | 15:22:21 |
| 22 Feb 2025 |
 hexa | I do see a number of builds that are stale (running for ~14h) and have zero log | 13:53:06 |
| hexa | killed, restarted and they are again running without a log | 14:20:46 |
| hexa | but I can see the builds running on the builders | 14:21:19 |
 John Ericson | hexa: is that....a semi-reproduce after all, or still no reproduce? | 17:40:33 |
| hexa | I can kill and restart and it repros | 17:40:58 |
| John Ericson | OK, and just to be clear this is builder hanging, no log, no progress | 17:41:37 |
| John Ericson | I've made git log master --first-parent very bisectable for this situation | 17:42:15 |
| John Ericson | though, I suspect the issue is the most recent merge | 17:42:23 |
| John Ericson | git log master --first-parent $(git merge-base c3b6e7b master)...master to show what's been h.n.o and master | 17:43:27 |
| John Ericson | have time to deploy one of those other ones in the middle? | 17:45:02 |
| John Ericson | * have time to test out one of those other ones in the middle hexa? | 17:45:15 |
| hexa | Later tonight | 17:45:59 |
| hexa | In 3h+ | 17:46:07 |
| John Ericson | ok sounds good, thanks! | 17:46:14 |
| John Ericson | hopefully once we find the PR which broke it, I can write a test :) | 17:46:42 |
|  ghpzin joined the room. | 19:15:02 |
|  Josh Heinrichs joined the room. | 21:28:18 |
| 23 Feb 2025 |
| hexa | John Ericson: bisected to https://github.com/NixOS/hydra/commit/4a4a0f901c70676ee47f830d2ff6a72789ba1baf | 04:42:50 |
| John Ericson | @hexa:lossy.network: thanks Hexa! That's just the one I thought it would be | 07:07:02 |
| 24 Feb 2025 |
 hacker1024 | We have an x86_64 machine running Hydra and an aarch64 builder. On recent versions of Nix/Hydra (I've tried the one with Nix 2.25 pre-LegacySSHStore and Nix 2.26 post-LegacySSHStore, it looks like x86_64 jobs that depend on outputs built on the aarch64 machine (e.g. deployment scripts that use aarch64 system closures) are getting stuck indefinitely on "Sending inputs", and cannot even be cancelled.
Pure x86_64 and aarch64 jobs are still fine.
Has anyone had this too?
| 02:54:28 |
| hacker1024 | * We have an x86_64 machine running Hydra and an aarch64 builder. On recent versions of Nix/Hydra (I've tried the one with Nix 2.25 pre-LegacySSHStore and Nix 2.26 post-LegacySSHStore), it looks like x86_64 jobs that depend on outputs built on the aarch64 machine (e.g. deployment scripts that use aarch64 system closures) are getting stuck indefinitely on "Sending inputs", and cannot even be cancelled.
Pure x86_64 and aarch64 jobs are still fine.
Has anyone had this too?
| 02:54:40 |
| hacker1024 | * We have an x86_64 machine running Hydra and an aarch64 builder. On recent versions of Nix/Hydra (I've tried the one with Nix 2.25 pre-LegacySSHStore and Nix 2.26 post-LegacySSHStore), it looks like x86_64 jobs that depend on outputs built on the aarch64 machine (e.g. deployment scripts that use aarch64 system closures) are getting stuck indefinitely on "Sending inputs", and cannot even be cancelled.
Nothing significant seems to be getting logged on either machine.
Pure x86_64 and aarch64 jobs are still fine.
Has anyone had this too?
| 02:54:57 |
