| 16 Feb 2025 |
|  soopyc (she/it) changed their profile picture. | 09:33:10 |
| 19 Feb 2025 |
 hacker1024 | Is there a straightforward way to clear the record of previously defined jobsets and previously built jobs? i.e. basically start fresh, while keeping the VCS cache if possible | 23:16:14 |
| 21 Feb 2025 |
| soopyc (she/it) | hi, updating to hydra 18c0d76 seems to cause remote builder jobs to stall for an indefinite amount of time (new builds logged nothing in 18h).
pinning to 881462b let jobs build again so i suspect something broke in the migration to LegacySSHStore | 06:19:12 |
 hexa |
Hydra 0.1.20250218.18c0d76
| 14:29:12 |
| hexa | running with 24 jobs on my private hydra | 14:29:20 |
| hexa | soopyc (she/it): cannot reproduce | 14:29:34 |
| soopyc (she/it) | interesting, will try again | 14:29:55 |
| hexa | https://hydra.lossy.network/machines proof 😛 | 14:30:26 |
| soopyc (she/it) | wack, everything works now for some reason :p sorry for the false alarm; could very well be a temporary blip. | 15:22:21 |
| 22 Feb 2025 |
| hexa | I do see a number of builds that are stale (running for ~14h) and have zero log | 13:53:06 |
| hexa | killed, restarted and they are again running without a log | 14:20:46 |
| hexa | but I can see the builds running on the builders | 14:21:19 |
 John Ericson | hexa: is that....a semi-reproduce after all, or still no reproduce? | 17:40:33 |
| hexa | I can kill and restart and it repros | 17:40:58 |
| John Ericson | OK, and just to be clear this is builder hanging, no log, no progress | 17:41:37 |
| John Ericson | I've made git log master --first-parent very bisectable for this situation | 17:42:15 |
| John Ericson | though, I suspect the issue is the most recent merge | 17:42:23 |
| John Ericson | git log master --first-parent $(git merge-base c3b6e7b master)...master to show what's been h.n.o and master | 17:43:27 |
| John Ericson | have time to deploy one of those other ones in the middle? | 17:45:02 |
| John Ericson | * have time to test out one of those other ones in the middle hexa? | 17:45:15 |
| hexa | Later tonight | 17:45:59 |
| hexa | In 3h+ | 17:46:07 |
| John Ericson | ok sounds good, thanks! | 17:46:14 |
| John Ericson | hopefully once we find the PR which broke it, I can write a test :) | 17:46:42 |
|  ghpzin (moved to @ghpzin:envs.net) joined the room. | 19:15:02 |
|  Josh Heinrichs joined the room. | 21:28:18 |
| 23 Feb 2025 |
| hexa | John Ericson: bisected to https://github.com/NixOS/hydra/commit/4a4a0f901c70676ee47f830d2ff6a72789ba1baf | 04:42:50 |
| John Ericson | @hexa:lossy.network: thanks Hexa! That's just the one I thought it would be | 07:07:02 |
| 24 Feb 2025 |
| hacker1024 | We have an x86_64 machine running Hydra and an aarch64 builder. On recent versions of Nix/Hydra (I've tried the one with Nix 2.25 pre-LegacySSHStore and Nix 2.26 post-LegacySSHStore, it looks like x86_64 jobs that depend on outputs built on the aarch64 machine (e.g. deployment scripts that use aarch64 system closures) are getting stuck indefinitely on "Sending inputs", and cannot even be cancelled.
Pure x86_64 and aarch64 jobs are still fine.
Has anyone had this too?
| 02:54:28 |
| hacker1024 | * We have an x86_64 machine running Hydra and an aarch64 builder. On recent versions of Nix/Hydra (I've tried the one with Nix 2.25 pre-LegacySSHStore and Nix 2.26 post-LegacySSHStore), it looks like x86_64 jobs that depend on outputs built on the aarch64 machine (e.g. deployment scripts that use aarch64 system closures) are getting stuck indefinitely on "Sending inputs", and cannot even be cancelled.
Pure x86_64 and aarch64 jobs are still fine.
Has anyone had this too?
| 02:54:40 |
