| 2 Mar 2024 |
 delroth | the cost on AWS is in large part bandwidth | 22:34:45 |
 raitobezarius | (80TB/month) | 22:34:51 |
| delroth | I don't have the exact breakdown, but $thousands/month | 22:35:14 |
| raitobezarius | I think it's 3K-ish | 22:35:22 |
| raitobezarius | OTOH | 22:35:26 |
| delroth | I thought it was more | 22:35:29 |
| raitobezarius | maybe I'm wrong, I don't have access to cost explorer | 22:35:38 |
| raitobezarius | I'm playing all those values by mind | 22:35:43 |
| raitobezarius | but then I think we can say ≥ 3K | 22:35:49 |
| delroth | In reply to @raitobezarius:matrix.org
maybe I'm wrong, I don't have access to cost explorer I have access to cost explorer but I don't think I know how to interpret it - I'm pretty sure the $9K/month discount is applied weirdly in there | 22:37:34 |
| raitobezarius | mmm, I remember we were able to isolate the bandwidth costs without the 9K/mo there but yeah | 22:37:57 |
| raitobezarius | during a meeting with zimbatm with the S3 folks | 22:38:06 |
 nh2 | In reply to @delroth:delroth.net
(AFAIK nobody has made a proper call on what kind of availability target we'd like to hit, so it's hard to know what kind of HA requirements as well as staffing we'd need) For comparison, my startup is 3 people. We run a Ceph cluster for 4 years, backing a public-facing product, and the Ceph has higher uptime / availablity than S3 in those 4 years.
In practice, only I maintain the cluster, and spend less than 2% of my working time on it. | 22:38:39 |
| delroth | 
Download out.png | 22:42:38 |
| delroth | possibly pre-tax | 22:42:54 |
| delroth | so might need to multiply by 1.21 | 22:43:01 |
| raitobezarius | pre-tax | 22:43:01 |
| raitobezarius | tax events happen at "point in time" | 22:43:06 |
| delroth | (that's Jan 2024) | 22:43:14 |
| raitobezarius | (and cause a spike in the cost explorer IIRC) | 22:43:17 |
| delroth | so if we could spend e.g. $250/month on reducing those $3500 down to $500 or so... I think we'd have a lot of space to experiment with how to handle the cold storage | 22:55:50 |
| delroth | and I don't think it's a significant effort to get there | 22:56:03 |
| delroth | well, it's definitely a few months of planning and work | 22:56:20 |
| delroth | but it's maybe 10x less effort than a full migration away from S3? | 22:57:04 |
| delroth | also postpones solving the availability problem since you can failover | 22:57:40 |
| delroth | (assuming we keep dual-writing) | 22:57:49 |
| delroth | * (assuming we keep dual-writing, which we should) | 22:58:07 |
| nh2 | In reply to @delroth:delroth.net
so if we could spend e.g. $250/month on reducing those $3500 down to $500 or so... I think we'd have a lot of space to experiment with how to handle the cold storage That's a good point. Make a cheaper cluster for serving, where the costs of the additional cluster (traffic + storage) are cheaper than the S3 traffic money saved. | 23:02:20 |
| delroth | another benefit is that we can use that new cache infra as an on-ramp into the more complex full S3 migration | 23:03:04 |
| delroth | if we build it roughly how we'd build the final stack except smaller | 23:03:26 |
