| 2 Mar 2024 |
 delroth | (that's Jan 2024) | 22:43:14 |
 raitobezarius | (and cause a spike in the cost explorer IIRC) | 22:43:17 |
| delroth | so if we could spend e.g. $250/month on reducing those $3500 down to $500 or so... I think we'd have a lot of space to experiment with how to handle the cold storage | 22:55:50 |
| delroth | and I don't think it's a significant effort to get there | 22:56:03 |
| delroth | well, it's definitely a few months of planning and work | 22:56:20 |
| delroth | but it's maybe 10x less effort than a full migration away from S3? | 22:57:04 |
| delroth | also postpones solving the availability problem since you can failover | 22:57:40 |
| delroth | (assuming we keep dual-writing) | 22:57:49 |
| delroth | * (assuming we keep dual-writing, which we should) | 22:58:07 |
 nh2 | In reply to @delroth:delroth.net
so if we could spend e.g. $250/month on reducing those $3500 down to $500 or so... I think we'd have a lot of space to experiment with how to handle the cold storage That's a good point. Make a cheaper cluster for serving, where the costs of the additional cluster (traffic + storage) are cheaper than the S3 traffic money saved. | 23:02:20 |
| delroth | another benefit is that we can use that new cache infra as an on-ramp into the more complex full S3 migration | 23:03:04 |
| delroth | if we build it roughly how we'd build the final stack except smaller | 23:03:26 |
| delroth | which has a lot of value in making people feel more comfortable with maintaining the infra long term, getting familiar with the tech stack, etc. | 23:03:52 |
| 3 Mar 2024 |
 Jonas Chevalier | side note, but nh2: you might also be interested in joining the archivist channel with flokli and edef, where it's more about content-deduplication: #archivists:nixos.org . the volume is quite slow, but feel free to ask them questions. | 12:08:27 |
| Jonas Chevalier | the main thing that is holing the dedupe back is hitting the AWS pricing, so moving most or everything to self-hosted would make things easier | 12:09:14 |
| Jonas Chevalier | raitobezarius: I'm not too concerned about a sustained 2k/month cost. I know at least one company willing to sponsor 100k / year. The only thing that is missing is for the foundation to approach them with a proposal that makes sense. And I think there are more companies out there like that. | 12:11:40 |
| Jonas Chevalier | we've been circling this particular issue for a while now. let's take the plunge. especially if nh2 is willing to help set things up and teach us about Ceph. we could order 3 servers and get a prototype up and running. | 12:13:49 |
| raitobezarius | In reply to @zimbatm:numtide.com
raitobezarius: I'm not too concerned about a sustained 2k/month cost. I know at least one company willing to sponsor 100k / year. The only thing that is missing is for the foundation to approach them with a proposal that makes sense. And I think there are more companies out there like that. If we are confident in this, I agree :) | 13:12:08 |
| delroth | In reply to @zimbatm:numtide.com
we've been circling this particular issue for a while now. let's take the plunge. especially if nh2 is willing to help set things up and teach us about Ceph. we could order 3 servers and get a prototype up and running. by "let's take the plunge" and "teach us", who is "us"? Because even if we have a Ceph cluster running, without the required work to actually use it to displace AWS usage, it's just an extra liability. And imo that work is more involved than the work required to set up a Ceph cluster, if only because nobody has even charted in details what needs to be done... | 22:15:14 |
| delroth | In reply to @zimbatm:numtide.com
we've been circling this particular issue for a while now. let's take the plunge. especially if nh2 is willing to help set things up and teach us about Ceph. we could order 3 servers and get a prototype up and running. * by "let's take the plunge" and "teach us", who do you mean by "us"? Because even if we have a Ceph cluster running, without the required work to actually use it to displace AWS usage, it's just an extra liability. And imo that work is more involved than the work required to set up a Ceph cluster, if only because nobody has even charted in details what needs to be done... | 22:15:33 |
| delroth | tl;dr: yes, at some point we need a Ceph cluster, but before we do the fun expensive part of setting up new shiny infra, we need someone to figure out the boring work of enabling Hydra to dual-write, enabling Fastly to dual-read, figuring out how we GC stuff from our new "hot cache" (if we go that route), etc. - all that is free, can be done now, and nobody has been lining up to do that work :) | 22:18:29 |
| raitobezarius | Hydra dual-write is on my todo list | 22:35:36 |
| raitobezarius | And the tooling developed for the S3 GC might be reapplicable for the "hot cache" GC, even a simple LRU seems a good start (?), better policies can be deployed as things happen | 22:36:52 |
| raitobezarius | but I mostly agree with you delroth anyway | 22:36:56 |
| raitobezarius | Fastly dual-read seems something that can be done only by people having accesses after reading all the docs and ensuring the answer is not already there as far as I understand it | 22:37:35 |
| raitobezarius | (and I believe we concluded by 'yes' in the deduplication meetings, a fallback path can be implemented) | 22:37:53 |
| raitobezarius | (as with Hydra dual-sign ahem) | 22:38:51 |
| delroth | In reply to @raitobezarius:matrix.org
Fastly dual-read seems something that can be done only by people having accesses after reading all the docs and ensuring the answer is not already there as far as I understand it it's not | 22:43:01 |
| delroth | I'm the person who did the last major Fastly change for cache.nixos.org and I did 90% of the work with no access, fwiw | 22:43:38 |
| raitobezarius | well I imagine that you either read the documentation or did it on your own account | 22:43:53 |
