| 2 Mar 2024 |
 raitobezarius | I'm playing all those values by mind | 22:35:43 |
| raitobezarius | but then I think we can say ≥ 3K | 22:35:49 |
 @delroth:delroth.net | In reply to @raitobezarius:matrix.org
maybe I'm wrong, I don't have access to cost explorer I have access to cost explorer but I don't think I know how to interpret it - I'm pretty sure the $9K/month discount is applied weirdly in there | 22:37:34 |
| raitobezarius | mmm, I remember we were able to isolate the bandwidth costs without the 9K/mo there but yeah | 22:37:57 |
| raitobezarius | during a meeting with zimbatm with the S3 folks | 22:38:06 |
 nh2 | In reply to @delroth:delroth.net
(AFAIK nobody has made a proper call on what kind of availability target we'd like to hit, so it's hard to know what kind of HA requirements as well as staffing we'd need) For comparison, my startup is 3 people. We run a Ceph cluster for 4 years, backing a public-facing product, and the Ceph has higher uptime / availablity than S3 in those 4 years.
In practice, only I maintain the cluster, and spend less than 2% of my working time on it. | 22:38:39 |
| @delroth:delroth.net | 
Download out.png | 22:42:38 |
| @delroth:delroth.net | possibly pre-tax | 22:42:54 |
| @delroth:delroth.net | so might need to multiply by 1.21 | 22:43:01 |
| raitobezarius | pre-tax | 22:43:01 |
| raitobezarius | tax events happen at "point in time" | 22:43:06 |
| @delroth:delroth.net | (that's Jan 2024) | 22:43:14 |
| raitobezarius | (and cause a spike in the cost explorer IIRC) | 22:43:17 |
| @delroth:delroth.net | so if we could spend e.g. $250/month on reducing those $3500 down to $500 or so... I think we'd have a lot of space to experiment with how to handle the cold storage | 22:55:50 |
| @delroth:delroth.net | and I don't think it's a significant effort to get there | 22:56:03 |
| @delroth:delroth.net | well, it's definitely a few months of planning and work | 22:56:20 |
| @delroth:delroth.net | but it's maybe 10x less effort than a full migration away from S3? | 22:57:04 |
| @delroth:delroth.net | also postpones solving the availability problem since you can failover | 22:57:40 |
| @delroth:delroth.net | (assuming we keep dual-writing) | 22:57:49 |
| @delroth:delroth.net | * (assuming we keep dual-writing, which we should) | 22:58:07 |
| nh2 | In reply to @delroth:delroth.net
so if we could spend e.g. $250/month on reducing those $3500 down to $500 or so... I think we'd have a lot of space to experiment with how to handle the cold storage That's a good point. Make a cheaper cluster for serving, where the costs of the additional cluster (traffic + storage) are cheaper than the S3 traffic money saved. | 23:02:20 |
| @delroth:delroth.net | another benefit is that we can use that new cache infra as an on-ramp into the more complex full S3 migration | 23:03:04 |
| @delroth:delroth.net | if we build it roughly how we'd build the final stack except smaller | 23:03:26 |
| @delroth:delroth.net | which has a lot of value in making people feel more comfortable with maintaining the infra long term, getting familiar with the tech stack, etc. | 23:03:52 |
| 3 Mar 2024 |
 Jonas Chevalier | side note, but nh2: you might also be interested in joining the archivist channel with flokli and edef, where it's more about content-deduplication: #archivists:nixos.org . the volume is quite slow, but feel free to ask them questions. | 12:08:27 |
| Jonas Chevalier | the main thing that is holing the dedupe back is hitting the AWS pricing, so moving most or everything to self-hosted would make things easier | 12:09:14 |
| Jonas Chevalier | raitobezarius: I'm not too concerned about a sustained 2k/month cost. I know at least one company willing to sponsor 100k / year. The only thing that is missing is for the foundation to approach them with a proposal that makes sense. And I think there are more companies out there like that. | 12:11:40 |
| Jonas Chevalier | we've been circling this particular issue for a while now. let's take the plunge. especially if nh2 is willing to help set things up and teach us about Ceph. we could order 3 servers and get a prototype up and running. | 12:13:49 |
| raitobezarius | In reply to @zimbatm:numtide.com
raitobezarius: I'm not too concerned about a sustained 2k/month cost. I know at least one company willing to sponsor 100k / year. The only thing that is missing is for the foundation to approach them with a proposal that makes sense. And I think there are more companies out there like that. If we are confident in this, I agree :) | 13:12:08 |
| @delroth:delroth.net | In reply to @zimbatm:numtide.com
we've been circling this particular issue for a while now. let's take the plunge. especially if nh2 is willing to help set things up and teach us about Ceph. we could order 3 servers and get a prototype up and running. by "let's take the plunge" and "teach us", who is "us"? Because even if we have a Ceph cluster running, without the required work to actually use it to displace AWS usage, it's just an extra liability. And imo that work is more involved than the work required to set up a Ceph cluster, if only because nobody has even charted in details what needs to be done... | 22:15:14 |
