| 28 Jun 2025 |
 hexa | h.n.o has that as well | 03:32:27 |
 ElvishJerricco | My other builder has "disabledUntil" : 0, | 03:32:40 |
| hexa | yeah, if it never failed | 03:32:46 |
| hexa | I think consecutiveFailures gets reset when it continues working | 03:33:05 |
| hexa | but disabledUntil and lastFailure are sticky until queue-runner restart | 03:33:24 |
| hexa | possibly stale ssh session? | 03:33:51 |
| hexa | kill the local nix-daemon? | 03:34:04 |
| hexa | test the ssh connection? | 03:34:08 |
| ElvishJerricco | well ssh is working, it seems | 03:36:08 |
| ElvishJerricco | the way this hydra is set up is a little stupid | 03:36:17 |
| ElvishJerricco | it's running in a nixos container on the host that is the builder :P | 03:36:34 |
| ElvishJerricco | I don't remember why I set it up this way; I assume the daemon-socket bind mounted from the host in the container would have let hydra use the local machine as a builder | 03:38:23 |
| ElvishJerricco | huh, restarting the nix daemon on the host, there's three processes left in the systemd unit from the previous service instance | 03:40:06 |
| ElvishJerricco | I think whatever's going on has something to do with failed builds. It seems like it's chewing through successful builds, but once one fails it stops scheduling builds and never starts again | 03:47:50 |
| ElvishJerricco | I recently updated this system, which included this hydra update: https://github.com/NixOS/nixpkgs/commit/cd9bf3369b9fc4ea0a6a8d91902a41d520580cb9
Which begins with this commit: https://github.com/NixOS/hydra/commit/720db63d52ebcbda617603e7aa5b5c750cc6afec | 05:05:32 |
| ElvishJerricco | hmmm | 05:05:40 |
| ElvishJerricco | mayhaps this "smarter scheduling criteria" is causing my problem? | 05:05:58 |
| ElvishJerricco | well, still seems like I have the problem after patching in a revert for that | 05:18:25 |
| ElvishJerricco | ok no, reverting hydra's src to the version I was using before does not fix the problem | 05:30:01 |
| ElvishJerricco | I'm at a loss here. Anybody have any ideas I can try? | 05:32:52 |
| 30 Jun 2025 |
|  jonhermansen joined the room. | 18:12:55 |
| ElvishJerricco | well, I tried deleting the whole Hydra project and remaking it in case it was some weird state bug | 18:40:29 |
| ElvishJerricco | same thing | 18:40:32 |
| 7 Jul 2025 |
|  @colemickens:matrix.org left the room. | 21:51:14 |
|  Katalin 🔪 changed their display name from Katalin ⚧︎ to Katalin 🔪. | 23:27:57 |
| 8 Jul 2025 |
| ElvishJerricco | well, my problem with the queue not continuing is apparently nothing stateful. I made a whole new container for it and recreated my jobsets, and it still does the same thing | 07:18:33 |
|  @janne.hess:helsinki-systems.de invited  Janne. | 21:21:03 |
| Janne joined the room. | 21:21:08 |
| @janne.hess:helsinki-systems.de left the room. | 21:22:06 |
| 13 Jul 2025 |
|  timschumi joined the room. | 18:29:56 |
