| 8 Oct 2021 |
 Jonas Chevalier | I added some templates to the repo and looking for feedback: https://github.com/NixOS/nixos-org-configurations/issues/new/choose
The goal is to make it easier for users to report issues with the infrastructure, and in the future request access or new resources to be deployed. | 07:33:59 |
| 10 Oct 2021 |
 hexa | There seems to be a caching issue with channels.nixos.org https://github.com/NixOS/nixos-org-configurations/issues/169#issuecomment-939478669 | 14:17:48 |
 sterni | this has been like this for over a week (two?) | 14:18:58 |
| sterni | seems like forever | 14:19:00 |
| sterni | also: has like any darwin build happened in the last week? | 14:48:22 |
 K900 | It feels like some of the CDN nodes are being weirdn | 15:03:57 |
| K900 | * It feels like some of the CDN nodes are being weird | 15:04:00 |
| K900 | I've been hitting really old builds for a while but it seems OK now | 15:04:10 |
| 11 Oct 2021 |
|  chvp joined the room. | 06:45:43 |
|  Ryan Burns joined the room. | 07:58:59 |
| Ryan Burns | In reply to @sternenseemann:systemli.org
also: has like any darwin build happened in the last week? nope. correct me if I'm wrong but based on https://hydra.nixos.org/status every darwin builder has been stuck on the same job for going on 7 days now | 08:01:43 |
| Ryan Burns | In reply to @sternenseemann:systemli.org
also: has like any darwin build happened in the last week? * nope. correct me if I'm wrong but based on https://hydra.nixos.org/status every x86_64-darwin builder has been stuck on the same job for going on 7 days now | 08:02:08 |
 toonn | Hmm, looks like there's 7 idle x86_64-darwin builders rn even though the queue is massive ~100k. Why might this happen? | 11:54:00 |
 lukegb (he/him) | Possibly some stdenv jobs are stuck on a broken worker and blocking everything else? | 11:54:44 |
| toonn | Do the Darwin builders require baby sitting, because I'm willing to do that for a bit if it's necessary? | 11:54:45 |
 Andreas Schrägle | hm. on our private hydra, we just restart the queue runner regularly, because things get stuck. I assume that's less practical on hydra.nixos.org. | 11:58:18 |
 Vladimír Čunát | Yes it actually looks like everything is waiting on build steps that are in some kind of stuck state:
"x86_64-darwin" : {
"runnable" : 0,
"running" : 29
},
| 12:32:41 |
| Vladimír Čunát | i.e. I suspect that restarting the queue runner might indeed help by itself in this situation. | 12:33:33 |
 @grahamc:nixos.org | I can help with that, also I think I can bring up some more capacity today | 12:49:42 |
| @grahamc:nixos.org | sorry for being MIA, I haven't been feeling myself the past few weeks | 12:49:50 |
| Andreas Schrägle | In reply to @vcunat:matrix.org
i.e. I suspect that restarting the queue runner might indeed help by itself in this situation. maybe someone should try and fix the queue runner at some point. that's more of a topic for #hydra:nixos.org though. | 12:50:58 |
 Rick (Mindavi) | I saw some improvements on the queue runner / event system recently, which may help | 13:06:38 |
| @grahamc:nixos.org | I doubt it -- that work will help with other issues not really seen often on hydra.nixos.org | 14:10:24 |
| @grahamc:nixos.org | (esp. around declarative jobsets and other plugins that expect read-after-write from their cache) | 14:12:58 |
|  niksnut joined the room. | 14:20:18 |
| niksnut | I don't see a lot of interesting things in the queue runner log, except that mac1, mac6, mac8 and mac9 are unreachable or don't accept our key | 14:21:24 |
| niksnut | also root@147.75.32.151 ("bigmac") is giving an error about the host key changing | 14:22:25 |
| 12 Oct 2021 |
| Vladimír Čunát | Apparently some of the x86_64-darwin machines immediately got stuck again, in "Sending inputs" phase. | 06:30:54 |
| Vladimír Čunát | So again, we seem to be long-term stuck in state with relatively long queue but no runnable step (for this platform). | 06:32:31 |
 andi- | I'd love to know what the bottlneck on the infra team side here is. Is it hardware? Is it time? Is it lack of interest (in that platform)? What can be done from the wider community to support your work? Should we give up on darwin? | 12:46:05 |
