| 31 Aug 2021 |
 Jonas Chevalier | switching channels.nixos.org to Fastly | 09:16:34 |
| Jonas Chevalier | looking good | 09:19:42 |
 Domen Kožar | Nice! | 10:11:07 |
| 2 Sep 2021 |
|  garbas joined the room. | 18:49:08 |
| 3 Sep 2021 |
 andi- | Unless I am seeing this wrong we are down to one x86_64 builder from packet. That might also contribute to the slowness of hydra in the last few days. It is about 11 jobs running right now in total with hundreds runnnable. | 01:50:50 |
 @grahamc:nixos.org | taking a look | 01:57:53 |
| @grahamc:nixos.org | I'm seeing some weirdness from upstream in Equinix Metal and I've got a ticket open with their overnight staff | 02:55:42 |
| @grahamc:nixos.org | should be coming back online after some netops magic tricks | 03:10:52 |
| andi- | It is running more builds again. Yay <3 Thanks Graham! | 03:22:38 |
| Jonas Chevalier | switching releases.nixos.org to Fastly in a few | 06:57:46 |
| Jonas Chevalier | CORS is now enabled on all the Fastly services | 08:28:02 |
 Vladimír Čunát | Uh, what's the matter with this bunch of various builder errors?
https://hydra.nixos.org/build/151780623#tabs-buildsteps | 09:42:25 |
| @grahamc:nixos.org | that can happen if Equinix is takingor replacing spot instances, between the moments of our automationrefreshing keys and builders | 11:15:03 |
| Domen Kožar | I/O on hydra is really slow, it is copying 100MB ghc for over 2 minutes now | 14:59:50 |
| Domen Kožar | https://hydra.nixos.org/build/151962685 | 14:59:57 |
| Domen Kožar | actually it's more like 4 minutes | 15:00:19 |
| Domen Kožar | grahamc (he/him): just a note how much avoiding copying to hydra would help :) | 15:01:35 |
| @grahamc:nixos.org | oh yeah I know | 15:01:51 |
| Domen Kožar | 5min.. | 15:01:53 |
| Domen Kožar | wait, does hydra do the compression or the builder? | 15:02:31 |
| @grahamc:nixos.org | https://monitoring.nixos.org/prometheus/graph?g0.expr=rate(node_network_receive_bytes_total%7Binstance%3D%22ceres%3A9100%22%7D%5B5m%5D)&g0.tab=0&g0.stacked=0&g0.range_input=1h this is in bytes per second | 15:03:12 |
| Domen Kožar | so slow! | 15:03:53 |
| @grahamc:nixos.org | slow? | 15:04:08 |
| @grahamc:nixos.org | yeah maybe a bit | 15:04:28 |
| Domen Kožar | 20MB/s? | 15:04:30 |
| Domen Kožar | or is that mbit? | 15:05:18 |
| Domen Kožar | ah, it's bytes, well still could be 5x on such a machine | 15:05:46 |
| @grahamc:nixos.org | yeah, there are a number of performance improvements that could probably be made | 15:06:12 |
| Domen Kožar | it's probably due to compression taking so much of the CPU | 15:06:39 |
| Domen Kožar | it's copying now for 10min | 15:07:04 |
