| 11 Mar 2024 |
 edef | i don't have stats on the build farm utilisation, maybe the infra team does | 17:36:31 |
 Shalok Shalom | different opinions about that from different sides | 17:36:36 |
| Shalok Shalom | I heard the word 'clusterfuck' 😃 | 17:37:01 |
| edef | on priors i'd lean towards "we just don't have enough compute" rather than "we're leaving it idle", but talk is cheap, i'd love to see some data | 17:37:11 |
| edef | like, broadly it's not very complicated, we (hopefully) have a Prometheus somewhere and a Grafana dash somewhere that can tell us this | 17:37:39 |
| Shalok Shalom | I know there is a Grafana | 17:38:07 |
| Shalok Shalom | didnt look into it too deep | 17:38:12 |
| Shalok Shalom | https://status.nixos.org/ | 17:38:23 |
| edef | the build scheduler definitely has issues, and we could do a lot better | 17:38:41 |
| edef | i certainly have some prototype bits and pieces lying around for various parts of the stack, but none of that is productionised or tested at scale | 17:39:14 |
| edef | i would be quite surprised if we are below 50% utilisation and have the capacity to run a second, identical workload | 17:39:41 |
| Shalok Shalom | yeah, I see that | 17:39:59 |
| Shalok Shalom | didnt intend to say, we should double the workload, just consider the testing in general | 17:40:26 |
| Shalok Shalom | I feel like sticking to something, just because its sticky, is also not worth necessarily | 17:40:48 |
| Shalok Shalom | * I feel like sticking to something, just because its sticky, is also not worth it. | 17:41:02 |
| edef | hmm, these dashes don't give me a lot of information directly, i'd have to actually work on building a decent model | 17:41:03 |
| Shalok Shalom | In reply to @edef1c:matrix.org
hmm, these dashes don't give me a lot of information directly, i'd have to actually work on building a decent model oh, that would be lovely | 17:41:28 |
| Shalok Shalom | you mean a model that is representative of the actual workload, yes? | 17:41:45 |
| edef | all in all, Hydra is pretty Lindy, we've been running it for far over a decade now | 17:41:58 |
| edef | no credible replacement has that kind of track record, and we have a lot to lose if we screw up | 17:42:13 |
| Shalok Shalom | yeah, I think its uniqueness is the reason to this scenario | 17:42:45 |
| edef | like, as i said, we are a long way from rock bottom, and i'd prefer not to go there | 17:42:59 |
| edef | Hydra does a fair few things that are plainly stupid, but hobbled along at lower scale and weren't worth the investment then | 17:43:44 |
| edef | now we are a fair bit further along the line, and it's time to pay the piper | 17:44:01 |
|  Dandellion joined the room. | 21:03:02 |
|  Claes joined the room. | 22:22:55 |
| 12 Mar 2024 |
|  samrose joined the room. | 08:23:51 |
|  h7x4 joined the room. | 08:32:44 |
| 14 Mar 2024 |
|  Federico Damián Schonborn left the room. | 02:04:28 |
|  NixOS Moderation Botchanged room power levels. | 18:44:17 |
