| 11 Oct 2021 |
 Andreas Schrägle | hm. on our private hydra, we just restart the queue runner regularly, because things get stuck. I assume that's less practical on hydra.nixos.org. | 11:58:18 |
 Vladimír Čunát | Yes it actually looks like everything is waiting on build steps that are in some kind of stuck state:
"x86_64-darwin" : {
"runnable" : 0,
"running" : 29
},
| 12:32:41 |
| Vladimír Čunát | i.e. I suspect that restarting the queue runner might indeed help by itself in this situation. | 12:33:33 |
 @grahamc:nixos.org | I can help with that, also I think I can bring up some more capacity today | 12:49:42 |
| @grahamc:nixos.org | sorry for being MIA, I haven't been feeling myself the past few weeks | 12:49:50 |
| Andreas Schrägle | In reply to @vcunat:matrix.org
i.e. I suspect that restarting the queue runner might indeed help by itself in this situation. maybe someone should try and fix the queue runner at some point. that's more of a topic for #hydra:nixos.org though. | 12:50:58 |
 Rick (Mindavi) | I saw some improvements on the queue runner / event system recently, which may help | 13:06:38 |
| @grahamc:nixos.org | I doubt it -- that work will help with other issues not really seen often on hydra.nixos.org | 14:10:24 |
| @grahamc:nixos.org | (esp. around declarative jobsets and other plugins that expect read-after-write from their cache) | 14:12:58 |
|  niksnut joined the room. | 14:20:18 |
| niksnut | I don't see a lot of interesting things in the queue runner log, except that mac1, mac6, mac8 and mac9 are unreachable or don't accept our key | 14:21:24 |
| niksnut | also root@147.75.32.151 ("bigmac") is giving an error about the host key changing | 14:22:25 |
| 12 Oct 2021 |
| Vladimír Čunát | Apparently some of the x86_64-darwin machines immediately got stuck again, in "Sending inputs" phase. | 06:30:54 |
| Vladimír Čunát | So again, we seem to be long-term stuck in state with relatively long queue but no runnable step (for this platform). | 06:32:31 |
 andi- | I'd love to know what the bottlneck on the infra team side here is. Is it hardware? Is it time? Is it lack of interest (in that platform)? What can be done from the wider community to support your work? Should we give up on darwin? | 12:46:05 |
 toonn | What seems weird is none of the jobs on the builders that are actually still building (3 out of 14) looks like a show-stopper. There's no stdenv jobs everything else could be waiting on for example. | 13:25:53 |
 lukegb (he/him) | Why is there a machine named # with type "Vladimir's" :P | 13:32:25 |
 hexa | Vladimír Čunát: 🙂 | 13:36:44 |
| hexa | * probably belongs to Vladimír Čunát | 13:37:15 |
| Vladimír Čunát | t??.cunat.cz machines are mine. (location, ownership, management) | 13:54:19 |
| Vladimír Čunát | It's really just one small machine at this moment only, though I hope to repair the one for metrics. | 13:54:58 |
| hexa | 
Download image.png | 13:55:11 |
| 13 Oct 2021 |
|  Piper McCorkle (she/her or they/them) joined the room. | 05:34:55 |
| Piper McCorkle (she/her or they/them) | What would the hurdles be to add qemu-binfmt to Hydra builders? I'm working on sparc64 and ppc64le bootstrap seeds and investigating the possibility of running the 3-stage build entirely on Hydra, so that they're actually just derived from the x86_64-linux seeds | 05:37:27 |
 Ryan Burns | The one thing I would worry about is poorly written cross packages "working" on hydra because of binfmt, because executable from the wrong arch are being run via qemu. It's good for the builders to mirror a "vanilla" NixOS install for the sake of local reproducibility. | 05:47:05 |
| Ryan Burns | This isn't an issue if it's possible to make binfmt opt-in on a per-package basis. Idk if that's possible or how hard that would be | 05:47:36 |
| Piper McCorkle (she/her or they/them) | Sadly you can't namespace binfmt definitions. Hmm... | 05:48:34 |
| Piper McCorkle (she/her or they/them) | I guess one way of namespacing would be do spin up an x86_64-linux NixOS VM with binfmt-misc inside, that way those systems don't need to have NixOS building | 05:54:42 |
| Ryan Burns | I wonder if it would be possible to do that and still get KVM accel | 05:56:30 |
| Piper McCorkle (she/her or they/them) | Should be, it's just a regular NixOS VM that happens to have binfmt_misc configured | 05:56:49 |
