| 6 Mar 2024 |
 edef | yeah, exactly | 17:21:36 |
| edef | one of the things in the works is incremental updating of the narinfo dataset from S3 | 17:21:50 |
 Wanja Hentze | and dedup, of course | 17:22:19 |
 Jonas Chevalier | if we can be smart about it, it has the potential to have better latency responses on the 404s | 17:22:24 |
| Jonas Chevalier | yeah and dedup will be easier to try, without worrying to pay 10k | 17:22:39 |
| Jonas Chevalier | * yeah and dedup will be easier to try, without worrying to pay 10k to AWS | 17:22:45 |
| edef | yeah. i have not worked on scaling deduplication compute much further because we didn't have the timeline to support that effort | 17:23:18 |
| edef | like. getting it into cold storage as-is is my current goal because i need more time budget, instead of frantically working to meet ever-shifting deletion deadlines | 17:24:09 |
| edef | as much as i am personally motivated to make this happen, this is a very stressful and strangely constrained working environment, and budget thus far would motivate no sane person to even get out of bed. i mostly do this because my sense of purpose is borderline anime protagonist levels | 17:27:44 |
 @patka_123:matrix.org | It's unasked for, but please take care of yourself | 17:29:10 |
| Wanja Hentze | make what exactly happen? self-hosted binary cache? or the more advanced stuff like dedup | 17:29:13 |
| edef | In reply to @zimbatm:numtide.com
One thing I am excited about if we end up self-hosting is that it will be easier to do experiments, like a smart narinfo database that can answer to queries. We can also introduce a log of all the new entries, to make it easier for other caches to mirror the main cache. i would like to have separate narinfo serving, and we are in a great position to do that. we can quite easily answer questions like "get me the entire closure of this path" in milliseconds now, if we want | 17:29:17 |
| edef | In reply to @whentze:matrix.org
make what exactly happen? self-hosted binary cache? or the more advanced stuff like dedup all of the above | 17:29:30 |
| edef | like, i'm not getting the past ~six months of my life back, my bank account nearly hit 0 doing this, other people have (probably wisely) dropped out because this is nuts and stressful | 17:31:44 |
| edef | having enthusiasm for self-hosting appear, nh2 show up to volunteer to help with Ceph, etc has been very good news to me | 17:32:20 |
| @patka_123:matrix.org | In reply to @edef1c:matrix.org
like, i'm not getting the past ~six months of my life back, my bank account nearly hit 0 doing this, other people have (probably wisely) dropped out because this is nuts and stressful Do you personally take donations? | 17:32:50 |
| edef | and if people offering funding follow through, especially long-term funding for hardware and hosting, that really changes the game | 17:33:16 |
| edef | In reply to @patka_123:matrix.org
Do you personally take donations? https://github.com/sponsors/edef1c is the default path, and any payment rail know to mankind can be arranged as well | 17:34:39 |
| edef | but i'm mostly not trying to advertise me or tell a sob story, i just hope we can actually follow through, seriously, with less churn in people | 17:35:52 |
| edef | eg flokli has been a great help and has done a bunch of the plumbing and edef-wrangling, but this has been quite hard on him | 17:36:44 |
| @patka_123:matrix.org | In reply to @edef1c:matrix.org
but i'm mostly not trying to advertise me or tell a sob story, i just hope we can actually follow through, seriously, with less churn in people I reached out for it myself, so it's not a sob story | 17:36:48 |
| edef | like, broadly the cache is in a weird spot because a) it's infrastructure. nobody gives a shit if you do your job right, and nothing has been deleted or noticeably broken yet b) it has been neglected for ages, this project should have started a years ago, not six months ago, and it shouldn't have depended on me as this much of a SPOF c) it's a public good. there is no value capture story for it, if you put money towards this, you will not gain reward in proportion. if anything, your net benefit from the cache work goes down if you spend a buck on it. so we have a free rider problem by nature, and thus far only Mercury has actually put up and shut up | 17:41:08 |
| edef | relative to the benefit received from it, i think the cache is well worth running and well worth running well. but the incentive structure and resulting actual budgets available are not a good time | 17:42:10 |
| edef | we didn't have data infrastructure for even seeing what was really in it or what it actually runs like until i wrote a bunch of it and coordinated with Jonas to get the AWS access i needed to do that, and while i love teaching people data tooling, data skills have been fairly SPOF-y so far | 17:44:16 |
| edef | if we can get meaningful, recurring funding so people who don't have as many screws loose are motivated to keep the show running, that's music to my ears | 17:47:54 |
| Wanja Hentze | is foundation funding even going towards this? | 17:50:34 |
