| 6 Mar 2024 |
 raitobezarius | In reply to @edef1c:matrix.org
i don't have empirical data to support that but i would treat that number with some skepticism I know durability loss in AWS | 17:09:43 |
| raitobezarius | In reply to @edef1c:matrix.org
i don't have empirical data to support that but i would treat that number with some skepticism * I know of durability loss stories in AWS | 17:09:48 |
| raitobezarius | And it was under the SLO | 17:09:57 |
| raitobezarius | (yada yada when you know how the sausage is made you know why this happens and in which scale, etC.) | 17:10:25 |
| raitobezarius | * (yada yada when you know how the sausage is made you know why this happens and in which scale, etc.) | 17:10:26 |
| raitobezarius | But I don't think we need to convince folks of the fact that SLO is a budget and you burn into it | 17:11:12 |
| raitobezarius | It's not a mathematical guarantee | 17:11:21 |
 edef | In reply to @edef1c:matrix.org
in general, throwing around numbers of nines is cute, but none of it means anything until you're specifying what SLOs these are for basically i don't really want to be all "you need to be this tall to ride, please have actual irl ops experience at scale to talk at all" but if people want to participate in conversations like this and make demands about SLOs/SLAs i'd like them to at least read the SRE at Google book and learn to think about this well, it's literally free | 17:11:27 |
 Jonas Chevalier | I'll talk to Domen and sort this out. I think he is motivated to raise funds, which could be pretty helpful. But yeah.. | 17:12:32 |
| edef | i would like to not lose a single bit of the cache, but i know it's statistically unlikely i can guarantee this; in expectation, we've already lost some | 17:12:33 |
| edef | but i do think we can buy a lot of latitude for mistakes on the serving/hot storage stack if we have a cold copy | 17:13:43 |
| edef | In reply to @raitobezarius:matrix.org
But I don't think we need to convince folks of the fact that SLO is a budget and you burn into it so like, one of the things i'm curious about is what the actual distributions are | 17:18:41 |
|  @blitz:chat.x86.lol joined the room. | 17:19:30 |
| edef | eg is this whole-object loss, object corruption, is this uncorrelated across objects or correlated within prefixes | 17:19:41 |
| Jonas Chevalier | One thing I am excited about if we end up self-hosting is that it will be easier to do experiments, like a smart narinfo database that can answer to queries. We can also introduce a log of all the new entries, to make it easier for other caches to mirror the main cache. | 17:21:24 |
| edef | yeah, exactly | 17:21:36 |
| edef | one of the things in the works is incremental updating of the narinfo dataset from S3 | 17:21:50 |
 Wanja Hentze | and dedup, of course | 17:22:19 |
| Jonas Chevalier | if we can be smart about it, it has the potential to have better latency responses on the 404s | 17:22:24 |
| Jonas Chevalier | yeah and dedup will be easier to try, without worrying to pay 10k | 17:22:39 |
| Jonas Chevalier | * yeah and dedup will be easier to try, without worrying to pay 10k to AWS | 17:22:45 |
| edef | yeah. i have not worked on scaling deduplication compute much further because we didn't have the timeline to support that effort | 17:23:18 |
| edef | like. getting it into cold storage as-is is my current goal because i need more time budget, instead of frantically working to meet ever-shifting deletion deadlines | 17:24:09 |
| edef | as much as i am personally motivated to make this happen, this is a very stressful and strangely constrained working environment, and budget thus far would motivate no sane person to even get out of bed. i mostly do this because my sense of purpose is borderline anime protagonist levels | 17:27:44 |
 patka | It's unasked for, but please take care of yourself | 17:29:10 |
