| 11 Mar 2024 |
 Shalok Shalom | In reply to @edef1c:matrix.org
scale is where ideas that look good on paper go to die, pretty much that is actually what I currently discuss with the dev of the other CI pipeline | 17:31:02 |
| Shalok Shalom | he hasnt tested such a scenario yet | 17:31:11 |
| Shalok Shalom | but indicates, that its possible | 17:31:22 |
 edef | like, i had all kinds of fun ideas for how quickly we would migrate the cache | 17:31:30 |
| Shalok Shalom | In reply to @whentze:matrix.org
and preferring an incremental replacement is not an endorsement of what's there thats a nice sentence 🙂 | 17:31:40 |
| Shalok Shalom | In reply to @edef1c:matrix.org
we could improve search queries and blob downloads quite directly yeah, the simple mind of mine thinks, it would have been done already, if it were so easy | 17:32:10 |
| Shalok Shalom | from the outside, it always looks very different than from the inside | 17:32:23 |
| edef | turns out that going from a few TB to 500TB, and a few million store paths to a quarter billion, a lot of things get weirder | 17:32:38 |
| Shalok Shalom | In reply to @edef1c:matrix.org
wherever migration paths do exist, they are generally not pre-tested on a system of this scale I would test the replacement side by side for a while
not decide if its worth it until its proven | 17:33:00 |
| Shalok Shalom | incremental improvements do have huge benefits because of that | 17:33:16 |
| Shalok Shalom | they turn out to provide quicker feedback | 17:33:30 |
| edef | a lot of the bare basics i one might have expected weren't there, and building them took a fair bit of blood, sweat, and tears | 17:33:34 |
| Shalok Shalom | In reply to @edef1c:matrix.org
turns out that going from a few TB to 500TB, and a few million store paths to a quarter billion, a lot of things get weirder but is it due to performance characteristics of Perl | 17:33:56 |
| edef | and yeah, yes to the above | 17:33:58 |
| Shalok Shalom | you mentioned the database and the backend | 17:34:10 |
| edef | but building a system that can run two things in parallel is still building a fair bit of new system | 17:34:29 |
| edef | In reply to @shalokshalom:dendrite.matrix.org
but is it due to performance characteristics of Perl i think Perl's perf should nowhere be on the hot path | 17:34:44 |
| Shalok Shalom | nice | 17:34:55 |
| edef | for the cache itself, there isn't really any perl on the hot path | 17:35:09 |
| Shalok Shalom | In reply to @edef1c:matrix.org
but building a system that can run two things in parallel is still building a fair bit of new system should be decoupled as far as possible | 17:35:15 |
| edef | we can't decouple so far that we're running two entirely parallel build pipelines | 17:35:38 |
| edef | we literally can't afford to run two build farms side by side | 17:35:45 |
| Shalok Shalom | Why is Hydra running new builds for the packages new to unstable only every 2 days? | 17:35:48 |
| Shalok Shalom | In reply to @edef1c:matrix.org
we literally can't afford to run two build farms side by side yeah, that is understandable. I would be tempted to care for that for the testing period. | 17:36:07 |
| edef | that's more about build farm capacity than anything about perl, i think | 17:36:11 |
| edef | i don't have stats on the build farm utilisation, maybe the infra team does | 17:36:31 |
| Shalok Shalom | different opinions about that from different sides | 17:36:36 |
| Shalok Shalom | I heard the word 'clusterfuck' 😃 | 17:37:01 |
| edef | on priors i'd lean towards "we just don't have enough compute" rather than "we're leaving it idle", but talk is cheap, i'd love to see some data | 17:37:11 |
| edef | like, broadly it's not very complicated, we (hopefully) have a Prometheus somewhere and a Grafana dash somewhere that can tell us this | 17:37:39 |
