| 7 Sep 2021 |
 Vladimír Čunát | That might hold for releases.nixos.org which contains the big stuff (and channels.nixos.org redirects there). | 09:35:28 |
 Jonas Chevalier | Yeah, it looks like releases.nixos.org is immutable/append only | 09:37:29 |
| Jonas Chevalier | waiting on feedback from the rest of the infra team for the fix: https://github.com/NixOS/nixos-channel-scripts/pull/49 | 10:14:11 |
 Domen Kožar | Jonas Chevalier ship it | 10:17:12 |
 @grahamc:nixos.org | Jonas Chevalier: I'm going to stash your changes to bastion so I can do a hydra deploy | 14:14:13 |
| Jonas Chevalier | I hit some nixops AWS issues updating the bastion, to deploy the channel update, so I moved that to Terraform: https://github.com/NixOS/nixos-org-configurations/pull/171 | 21:05:25 |
| 8 Sep 2021 |
|  mbprtpmnr joined the room. | 13:48:28 |
| Jonas Chevalier | deployed the channels-cache-control branch and waiting to see if it works as expected | 16:49:44 |
 lukegb (he/him) | https://hydra.nixos.org/build/151175520 hmm, this build looks sad | 22:58:17 |
| lukegb (he/him) | is mac2-guest ok? does it need a hug? | 22:58:26 |
 sterni | waiting for godot, upgrade: waiting for the big garbage collector lock... | 22:59:05 |
| lukegb (he/him) | ok yeah, that doesn't look healthy: https://p.lukegb.com/raw/StronglyAllowedOpossum.png | 23:02:29 |
 hexa | it's been like that for a while | 23:35:27 |
| hexa | * it's been like that for a while (few days) | 23:35:32 |
| 9 Sep 2021 |
| Domen Kožar | aarch64-darwin is now building staging, 21.05 and trunk | 06:43:15 |
| Domen Kožar | Bored most of the time :D | 06:43:24 |
| Vladimír Čunát | Oh, these stuck x86_64-darwin jobs were probably why the trunk jobset had about a hundred unfinished builds continuously and thus preventing the nixpkgs-unstable channel from advancing. | 06:44:41 |
| Vladimír Čunát | ... so now I just cancelled all older than three days. | 06:45:30 |
| Domen Kožar | good catch :) | 08:36:19 |
| lukegb (he/him) | Hmm, can we add x86_64-darwin as extra arch on the ARM builders or is that a bit broken because of the macOS version disparity | 10:28:56 |
| @grahamc:nixos.org | fwiw it doesn't look like a capacity issue at the moment, since there are macs that are able to take work but aren't given any | 11:37:00 |
| @grahamc:nixos.org | I think there is/was an alert on jobs that are taking a long time: https://monitoring.nixos.org/grafana/d/j0hJAY1Wk/in-progress-build-duration-heatmap?orgId=1&refresh=30s | 11:40:15 |
| @grahamc:nixos.org | yeah: https://monitoring.nixos.org/prometheus/alerts maybe we could get it posting to matrix | 11:40:37 |
| Vladimír Čunát | In reply to @grahamc:nixos.org
fwiw it doesn't look like a capacity issue at the moment, since there are macs that are able to take work but aren't given any Then something is wrong.
"x86_64-darwin" : {
"runnable" : 8939,
"running" : 55,
"waitTime" : 981988989
},
| 11:43:20 |
| @grahamc:nixos.org | yeah ... I have found runnable to be misleading ... but I don't remember the why ..... | 11:44:26 |
| @grahamc:nixos.org | I've done autoscaling based on runnable andended up running dozens of totally idle machines for days as a result | 11:47:59 |
| @grahamc:nixos.org | I have a client facing similar mysterious issues and wants to look in to them ~soon, but not yet | 11:51:24 |
 baloo | https://github.com/NixOS/nixpkgs/pull/137197 this should fix the dns resolution failure on hydra | 16:06:24 |
| @grahamc:nixos.org | Maybe we can get Eelco to do a release instead of apply patches | 16:10:19 |
| Vladimír Čunát | Well, it's relevant what nix gets deployed to the machines, not what we have in nixpkgs. | 16:10:27 |
