| 6 Mar 2024 |
 Jonas Chevalier | * yeah and dedup will be easier to try, without worrying to pay 10k to AWS | 17:22:45 |
 edef | yeah. i have not worked on scaling deduplication compute much further because we didn't have the timeline to support that effort | 17:23:18 |
| edef | like. getting it into cold storage as-is is my current goal because i need more time budget, instead of frantically working to meet ever-shifting deletion deadlines | 17:24:09 |
| edef | as much as i am personally motivated to make this happen, this is a very stressful and strangely constrained working environment, and budget thus far would motivate no sane person to even get out of bed. i mostly do this because my sense of purpose is borderline anime protagonist levels | 17:27:44 |
 patka | It's unasked for, but please take care of yourself | 17:29:10 |
 Wanja Hentze | make what exactly happen? self-hosted binary cache? or the more advanced stuff like dedup | 17:29:13 |
| edef | In reply to @zimbatm:numtide.com
One thing I am excited about if we end up self-hosting is that it will be easier to do experiments, like a smart narinfo database that can answer to queries. We can also introduce a log of all the new entries, to make it easier for other caches to mirror the main cache. i would like to have separate narinfo serving, and we are in a great position to do that. we can quite easily answer questions like "get me the entire closure of this path" in milliseconds now, if we want | 17:29:17 |
| edef | In reply to @whentze:matrix.org
make what exactly happen? self-hosted binary cache? or the more advanced stuff like dedup all of the above | 17:29:30 |
| edef | like, i'm not getting the past ~six months of my life back, my bank account nearly hit 0 doing this, other people have (probably wisely) dropped out because this is nuts and stressful | 17:31:44 |
| edef | having enthusiasm for self-hosting appear, nh2 show up to volunteer to help with Ceph, etc has been very good news to me | 17:32:20 |
| patka | In reply to @edef1c:matrix.org
like, i'm not getting the past ~six months of my life back, my bank account nearly hit 0 doing this, other people have (probably wisely) dropped out because this is nuts and stressful Do you personally take donations? | 17:32:50 |
| edef | and if people offering funding follow through, especially long-term funding for hardware and hosting, that really changes the game | 17:33:16 |
| edef | In reply to @patka_123:matrix.org
Do you personally take donations? https://github.com/sponsors/edef1c is the default path, and any payment rail know to mankind can be arranged as well | 17:34:39 |
| edef | but i'm mostly not trying to advertise me or tell a sob story, i just hope we can actually follow through, seriously, with less churn in people | 17:35:52 |
| edef | eg flokli has been a great help and has done a bunch of the plumbing and edef-wrangling, but this has been quite hard on him | 17:36:44 |
| patka | In reply to @edef1c:matrix.org
but i'm mostly not trying to advertise me or tell a sob story, i just hope we can actually follow through, seriously, with less churn in people I reached out for it myself, so it's not a sob story | 17:36:48 |
| edef | like, broadly the cache is in a weird spot because a) it's infrastructure. nobody gives a shit if you do your job right, and nothing has been deleted or noticeably broken yet b) it has been neglected for ages, this project should have started a years ago, not six months ago, and it shouldn't have depended on me as this much of a SPOF c) it's a public good. there is no value capture story for it, if you put money towards this, you will not gain reward in proportion. if anything, your net benefit from the cache work goes down if you spend a buck on it. so we have a free rider problem by nature, and thus far only Mercury has actually put up and shut up | 17:41:08 |
| edef | relative to the benefit received from it, i think the cache is well worth running and well worth running well. but the incentive structure and resulting actual budgets available are not a good time | 17:42:10 |
| edef | we didn't have data infrastructure for even seeing what was really in it or what it actually runs like until i wrote a bunch of it and coordinated with Jonas to get the AWS access i needed to do that, and while i love teaching people data tooling, data skills have been fairly SPOF-y so far | 17:44:16 |
| edef | if we can get meaningful, recurring funding so people who don't have as many screws loose are motivated to keep the show running, that's music to my ears | 17:47:54 |
| Wanja Hentze | is foundation funding even going towards this? | 17:50:34 |
| Wanja Hentze | a foundation is a way to fund a Commons, right? | 17:50:51 |
| edef | there is an opencollective thing for the cache sustainability/scalability/preservation project | 17:51:10 |
| Wanja Hentze | oh, nice | 17:51:32 |
| edef | but broadly speaking i am not impressed with how much the (seemingly profitable) ecosystem has contributed towards the Foundation | 17:51:46 |
| edef | the €5k i've asked for is certainly not enough to cover even my living costs for the amount of time i've sunk in, and i'm not even convinced i should be accepting that money, because the data and its availability isn't safe and sound just yet | 17:54:00 |
| edef | we have other infrastructural things to improve, Hydra could certainly use work, the build farm in general could use work, i'm just tending to the one that feels like the biggest emergency right now | 17:56:34 |
| edef | some of these other things will turn into emergencies in due time | 17:56:55 |
| edef | the cache is in rough seas because one sponsor dropped out and we had to hop to the next one, none of these are eternal, same goes for CDNs and build farm compute providers | 18:00:23 |
| edef | a bunch of the infrastructure governance thus far has been abysmal, and no shade to a bunch of the infra team there, they've been doing their best with what they have | 18:01:06 |
