| 20 May 2026 |
 hexa (signing key rotation when) | those are our linux builders, the first two x86_64-linux, the last two aarch64-linux | 11:57:39 |
 Cobalt | Thank you a lot! | 11:58:08 |
| hexa (signing key rotation when) | and our gc logic | 11:58:42 |
| Cobalt | Thank you for pointing me to the configurations, I was aware of that to some degree (searching for the metrics initially led me to the system configurations). The 500GB limit for a GC run is presumably to ensure that the GC is not locking the store for too long? | 12:01:36 |
| hexa (signing key rotation when) | yeah, gc is not free and if we keep some outputs around for reuse that's good too | 12:03:27 |
| hexa (signing key rotation when) | we might eventually migrate to nix-fast-gc in a bit | 12:03:59 |
| hexa (signing key rotation when) | that'll fix us | 12:04:12 |
| hexa (signing key rotation when) | Redacted or Malformed Event | 12:04:23 |
| hexa (signing key rotation when) | https://github.com/Mic92/fast-nix-gc | 12:04:35 |
| Cobalt | That also looks very interesting, especially the notes on SQLite queries in fast-nix-gc. | 12:06:42 |
 K900 | I'm not sure what kind of optimization you're looking at here, but generally GC or any kind of store queries aren't the bottleneck | 12:07:33 |
| Cobalt | The main optimizations were how to reduce the number/cost of queries for evaluating the subset size required when looking scheduling a set of builds. They might not be a bottlneck by themselves but caching and/or applicability of a stochastic data structure seems an interesting extension. My supervisor was interested in this specific sub-problem as it relates a bit to his own research iirc. | 12:13:23 |
| Cobalt | fast-nix-gc does not really have anything related to this, it just mentions that they load the paths into a graph for the GC search first instead of querying the store for all lookups. | 12:14:15 |
| K900 | Well right now the scheduling is very stupid | 12:14:20 |
| K900 | There's no locality awareness | 12:14:30 |
| K900 | Or hell job size awareness | 12:14:32 |
| K900 | Improving it will definitely help a little, but the big bottleneck is still the coordinator itself | 12:15:32 |
| hexa (signing key rotation when) | the scheduler | 12:15:50 |
| K900 | The coordinator as in the machine | 12:16:01 |
| K900 | But yeah | 12:16:02 |
| hexa (signing key rotation when) | the coordinator is the process that runs the remote build | 12:16:14 |
| Cobalt | So just to understand this a bit more, a significant problem is the performance of the software running the scheduler/coordinator (so the queue runner)? | 12:17:07 |
| K900 | It's not even the software necessarily | 12:18:34 |
| K900 | It's the design of the whole thing that requires a lot of copying data around | 12:18:44 |
| Cobalt | A montivation of the optimizations was to ensure that scheduling was supposed to stay cheap-ish so I would try to not compromise this too much. | 12:18:53 |
| K900 | And also the fact that everything is xz compressed in transport which is extremely overhead | 12:18:59 |
| Cobalt | Is that regarding the data exchange of the build outputs, RPC or artifacts (logs)? | 12:19:47 |
| K900 | Build outputs | 12:20:05 |
| K900 | Logs are negligible by comparison | 12:20:12 |
| K900 | And RPC is just normal Nix daemon protocol over SSH | 12:20:18 |
