| 3 Sep 2021 |
 Vladimír Čunát | Uh, what's the matter with this bunch of various builder errors?
https://hydra.nixos.org/build/151780623#tabs-buildsteps | 09:42:25 |
 @grahamc:nixos.org | that can happen if Equinix is takingor replacing spot instances, between the moments of our automationrefreshing keys and builders | 11:15:03 |
 Domen Kožar | I/O on hydra is really slow, it is copying 100MB ghc for over 2 minutes now | 14:59:50 |
| Domen Kožar | https://hydra.nixos.org/build/151962685 | 14:59:57 |
| Domen Kožar | actually it's more like 4 minutes | 15:00:19 |
| Domen Kožar | grahamc (he/him): just a note how much avoiding copying to hydra would help :) | 15:01:35 |
| @grahamc:nixos.org | oh yeah I know | 15:01:51 |
| Domen Kožar | 5min.. | 15:01:53 |
| Domen Kožar | wait, does hydra do the compression or the builder? | 15:02:31 |
| @grahamc:nixos.org | https://monitoring.nixos.org/prometheus/graph?g0.expr=rate(node_network_receive_bytes_total%7Binstance%3D%22ceres%3A9100%22%7D%5B5m%5D)&g0.tab=0&g0.stacked=0&g0.range_input=1h this is in bytes per second | 15:03:12 |
| Domen Kožar | so slow! | 15:03:53 |
| @grahamc:nixos.org | slow? | 15:04:08 |
| @grahamc:nixos.org | yeah maybe a bit | 15:04:28 |
| Domen Kožar | 20MB/s? | 15:04:30 |
| Domen Kožar | or is that mbit? | 15:05:18 |
| Domen Kožar | ah, it's bytes, well still could be 5x on such a machine | 15:05:46 |
| @grahamc:nixos.org | yeah, there are a number of performance improvements that could probably be made | 15:06:12 |
| Domen Kožar | it's probably due to compression taking so much of the CPU | 15:06:39 |
| Domen Kožar | it's copying now for 10min | 15:07:04 |
| Domen Kožar | 😱 | 15:07:27 |
| @grahamc:nixos.org | compression does happen at a variety of levels | 15:08:06 |
| Domen Kožar | ah, I meant xz for the binary cache | 15:08:25 |
| @grahamc:nixos.org | yeah, but it is actually probably worse | 15:08:32 |
| @grahamc:nixos.org | like we use SSH compression too | 15:08:36 |
| Domen Kožar | right | 15:08:49 |
| Domen Kožar | grahamc (he/him): how can we prioritize that patch of yours? :) | 15:09:08 |
| @grahamc:nixos.org | frankly, right now, finding someone to pay for the time to do it 😓 bootstrapping a business with a 1y/o makes it hard to get focused time on things like that otherwise | 15:10:02 |
| Domen Kožar | how much time is needed to shape it up? do you have a rough estimate? | 15:10:25 |
| @grahamc:nixos.org | I don't have an estimate really, it was extremely proof-of-concept | 15:10:45 |
| @grahamc:nixos.org | but I could try and come up with one | 15:10:51 |
