| 3 Sep 2021 |
 @grahamc:nixos.org | https://monitoring.nixos.org/prometheus/graph?g0.expr=rate(node_network_receive_bytes_total%7Binstance%3D%22ceres%3A9100%22%7D%5B5m%5D)&g0.tab=0&g0.stacked=0&g0.range_input=1h this is in bytes per second | 15:03:12 |
 Domen Kožar | so slow! | 15:03:53 |
| @grahamc:nixos.org | slow? | 15:04:08 |
| @grahamc:nixos.org | yeah maybe a bit | 15:04:28 |
| Domen Kožar | 20MB/s? | 15:04:30 |
| Domen Kožar | or is that mbit? | 15:05:18 |
| Domen Kožar | ah, it's bytes, well still could be 5x on such a machine | 15:05:46 |
| @grahamc:nixos.org | yeah, there are a number of performance improvements that could probably be made | 15:06:12 |
| Domen Kožar | it's probably due to compression taking so much of the CPU | 15:06:39 |
| Domen Kožar | it's copying now for 10min | 15:07:04 |
| Domen Kožar | 😱 | 15:07:27 |
| @grahamc:nixos.org | compression does happen at a variety of levels | 15:08:06 |
| Domen Kožar | ah, I meant xz for the binary cache | 15:08:25 |
| @grahamc:nixos.org | yeah, but it is actually probably worse | 15:08:32 |
| @grahamc:nixos.org | like we use SSH compression too | 15:08:36 |
| Domen Kožar | right | 15:08:49 |
| Domen Kožar | grahamc (he/him): how can we prioritize that patch of yours? :) | 15:09:08 |
| @grahamc:nixos.org | frankly, right now, finding someone to pay for the time to do it 😓 bootstrapping a business with a 1y/o makes it hard to get focused time on things like that otherwise | 15:10:02 |
| Domen Kožar | how much time is needed to shape it up? do you have a rough estimate? | 15:10:25 |
| @grahamc:nixos.org | I don't have an estimate really, it was extremely proof-of-concept | 15:10:45 |
| @grahamc:nixos.org | but I could try and come up with one | 15:10:51 |
| Domen Kožar | hmm | 15:11:42 |
| Domen Kožar | well if you come up with one that would help, I really think this would give us a huge boost | 15:12:00 |
| Domen Kožar | maybe just crowdfund it | 15:12:14 |
| Domen Kožar | savings are enormous, 100MB should compress in like 20s instead of 12min | 15:14:45 |
| @grahamc:nixos.org | I think it is a bit quick to jump to saying it is compression, but there is definitely a lot of benefit from moving cache uploads from central machine to the builder | 15:18:36 |
| Domen Kožar | it's I/O + compression | 15:19:03 |
| Domen Kožar | not sure what else it would be, the job was stuck in "receiving outputs" for 12min, that should otherwise be independent of hydra | 15:19:46 |
| @grahamc:nixos.org | well ... I wouldn't jump to that, personally, so soon -- but perhaps so :) | 15:20:14 |
| Domen Kožar | I reserve the right to be wrong! | 15:25:34 |
