| 3 Sep 2021 |
 @grahamc:nixos.org | this isn't a common situation really | 17:04:11 |
| @grahamc:nixos.org | the machines we get are usually very stable and long lived | 17:04:18 |
 Vladimír Čunát | Then it's OK, I think. Jobs need some babysitting anyway. | 17:04:49 |
| @grahamc:nixos.org | sometimes their spot market auction process gets in to a bad state and churns and churns | 17:05:02 |
| Vladimír Čunát | Black Friday today? | 17:07:54 |
 baloo | no it's in November | 17:08:47 |
| @grahamc:nixos.org | https://monitoring.nixos.org/grafana/d/wiaOmQ4nk/equinix-metal-churn?orgId=1&refresh=10s&var-instance_class=m1.xlarge.x86&var-facility=All | 17:20:39 |
| @grahamc:nixos.org | you can see the churn very clearly here | 17:20:56 |
| @grahamc:nixos.org | and one of the causes of the spot price in the top graph | 17:21:05 |
| @grahamc:nixos.org | ams1 isn't super happy either | 17:23:27 |
| @grahamc:nixos.org | I'll take both out | 17:23:30 |
| @grahamc:nixos.org | well now I've really stepped in it | 17:29:44 |
| @grahamc:nixos.org | recreating the spot market request deleted all the existing requests (I expected this) then a bug in either the terraform provider or their API made the "create" step fail | 17:42:37 |
| @grahamc:nixos.org | working with their support team | 18:05:02 |
| @grahamc:nixos.org | looks like we're getting hardware upgrades | 18:47:25 |
| @grahamc:nixos.org | I did not expect to spend half the work day on this 🙃 | 19:05:53 |
| 4 Sep 2021 |
| @grahamc:nixos.org | okay, I finished the work to get these new classes of hardware in hydra | 03:01:31 |
| @grahamc:nixos.org | went from machines with:
2x 2 x Intel Xeon E5-2650 v4 (ie: 24 cores, 2.2ghz) and 256G ram
to:
1x 1 x AMD EPYC 7402P (ie: 24 cores, 2.8ghz) and 64G RAM
and 1x 1 x AMD EPYC 7402P (ie: 24 cores, 2.8ghz) and 256G RAM | 03:03:12 |
| @grahamc:nixos.org | that second machine has 2x 25Gbps network connections :drool: | 03:03:41 |
| @grahamc:nixos.org | * went from machines with:
2x 2 x Intel Xeon E5-2650 v4 (ie: 24 cores, 2.2ghz) and 256G ram
to:
1x 1 x AMD EPYC 7402P (ie: 24 cores, 2.8ghz) and 64G RAM
and 1x 1 x AMD EPYC 7502P (ie: 32 cores @ 2.5Ghz) and 256G RAM | 03:05:43 |
 lukegb (he/him) | Ooh | 03:20:14 |
| lukegb (he/him) | Thanks so much for your work grahamc (he/him) ❤️ | 03:20:34 |
| @grahamc:nixos.org | :) | 03:22:29 |
| @grahamc:nixos.org | it is possible I broke the aarch64 machines while updating the channel | 03:43:04 |
| @grahamc:nixos.org | I'm not too stressed about that, but I'm also not likely to fix that tonight | 03:43:18 |
 Domen Kožar | nixpkgs-unstable is stuck for 6 days regardless of the builds passing | 09:03:22 |
| Vladimír Čunát | It had lots of unfinished jobs. | 09:22:09 |
| Vladimír Čunát | Even five days old evals still have unfinished jobs now. | 09:23:45 |
| Vladimír Čunát | x86_64-darwin is still overloaded, due to the openssl update about a week ago (causing stdenv rebuild in there unfortunately). | 09:24:23 |
 sterni | maybe time to make a separate nixpkgs-unstable-darwin channel as well? | 09:36:07 |
