| 6 Mar 2024 |
 Jonas Chevalier | I'll talk to Domen and sort this out. I think he is motivated to raise funds, which could be pretty helpful. But yeah.. | 17:12:32 |
 edef | i would like to not lose a single bit of the cache, but i know it's statistically unlikely i can guarantee this; in expectation, we've already lost some | 17:12:33 |
| edef | but i do think we can buy a lot of latitude for mistakes on the serving/hot storage stack if we have a cold copy | 17:13:43 |
| edef | In reply to @raitobezarius:matrix.org
But I don't think we need to convince folks of the fact that SLO is a budget and you burn into it so like, one of the things i'm curious about is what the actual distributions are | 17:18:41 |
|  @blitz:chat.x86.lol joined the room. | 17:19:30 |
| edef | eg is this whole-object loss, object corruption, is this uncorrelated across objects or correlated within prefixes | 17:19:41 |
| Jonas Chevalier | One thing I am excited about if we end up self-hosting is that it will be easier to do experiments, like a smart narinfo database that can answer to queries. We can also introduce a log of all the new entries, to make it easier for other caches to mirror the main cache. | 17:21:24 |
| edef | yeah, exactly | 17:21:36 |
| edef | one of the things in the works is incremental updating of the narinfo dataset from S3 | 17:21:50 |
 Wanja Hentze | and dedup, of course | 17:22:19 |
| Jonas Chevalier | if we can be smart about it, it has the potential to have better latency responses on the 404s | 17:22:24 |
| Jonas Chevalier | yeah and dedup will be easier to try, without worrying to pay 10k | 17:22:39 |
| Jonas Chevalier | * yeah and dedup will be easier to try, without worrying to pay 10k to AWS | 17:22:45 |
| edef | yeah. i have not worked on scaling deduplication compute much further because we didn't have the timeline to support that effort | 17:23:18 |
| edef | like. getting it into cold storage as-is is my current goal because i need more time budget, instead of frantically working to meet ever-shifting deletion deadlines | 17:24:09 |
| edef | as much as i am personally motivated to make this happen, this is a very stressful and strangely constrained working environment, and budget thus far would motivate no sane person to even get out of bed. i mostly do this because my sense of purpose is borderline anime protagonist levels | 17:27:44 |
 @patka_123:matrix.org | It's unasked for, but please take care of yourself | 17:29:10 |
| Wanja Hentze | make what exactly happen? self-hosted binary cache? or the more advanced stuff like dedup | 17:29:13 |
| edef | In reply to @zimbatm:numtide.com
One thing I am excited about if we end up self-hosting is that it will be easier to do experiments, like a smart narinfo database that can answer to queries. We can also introduce a log of all the new entries, to make it easier for other caches to mirror the main cache. i would like to have separate narinfo serving, and we are in a great position to do that. we can quite easily answer questions like "get me the entire closure of this path" in milliseconds now, if we want | 17:29:17 |
| edef | In reply to @whentze:matrix.org
make what exactly happen? self-hosted binary cache? or the more advanced stuff like dedup all of the above | 17:29:30 |
