| 2 Mar 2024 |
 @delroth:delroth.net | if we build it roughly how we'd build the final stack except smaller | 23:03:26 |
| @delroth:delroth.net | which has a lot of value in making people feel more comfortable with maintaining the infra long term, getting familiar with the tech stack, etc. | 23:03:52 |
| 3 Mar 2024 |
 Jonas Chevalier | side note, but nh2: you might also be interested in joining the archivist channel with flokli and edef, where it's more about content-deduplication: #archivists:nixos.org . the volume is quite slow, but feel free to ask them questions. | 12:08:27 |
| Jonas Chevalier | the main thing that is holing the dedupe back is hitting the AWS pricing, so moving most or everything to self-hosted would make things easier | 12:09:14 |
| Jonas Chevalier | raitobezarius: I'm not too concerned about a sustained 2k/month cost. I know at least one company willing to sponsor 100k / year. The only thing that is missing is for the foundation to approach them with a proposal that makes sense. And I think there are more companies out there like that. | 12:11:40 |
| Jonas Chevalier | we've been circling this particular issue for a while now. let's take the plunge. especially if nh2 is willing to help set things up and teach us about Ceph. we could order 3 servers and get a prototype up and running. | 12:13:49 |
 raitobezarius | In reply to @zimbatm:numtide.com
raitobezarius: I'm not too concerned about a sustained 2k/month cost. I know at least one company willing to sponsor 100k / year. The only thing that is missing is for the foundation to approach them with a proposal that makes sense. And I think there are more companies out there like that. If we are confident in this, I agree :) | 13:12:08 |
| @delroth:delroth.net | In reply to @zimbatm:numtide.com
we've been circling this particular issue for a while now. let's take the plunge. especially if nh2 is willing to help set things up and teach us about Ceph. we could order 3 servers and get a prototype up and running. by "let's take the plunge" and "teach us", who is "us"? Because even if we have a Ceph cluster running, without the required work to actually use it to displace AWS usage, it's just an extra liability. And imo that work is more involved than the work required to set up a Ceph cluster, if only because nobody has even charted in details what needs to be done... | 22:15:14 |
| @delroth:delroth.net | In reply to @zimbatm:numtide.com
we've been circling this particular issue for a while now. let's take the plunge. especially if nh2 is willing to help set things up and teach us about Ceph. we could order 3 servers and get a prototype up and running. * by "let's take the plunge" and "teach us", who do you mean by "us"? Because even if we have a Ceph cluster running, without the required work to actually use it to displace AWS usage, it's just an extra liability. And imo that work is more involved than the work required to set up a Ceph cluster, if only because nobody has even charted in details what needs to be done... | 22:15:33 |
| @delroth:delroth.net | tl;dr: yes, at some point we need a Ceph cluster, but before we do the fun expensive part of setting up new shiny infra, we need someone to figure out the boring work of enabling Hydra to dual-write, enabling Fastly to dual-read, figuring out how we GC stuff from our new "hot cache" (if we go that route), etc. - all that is free, can be done now, and nobody has been lining up to do that work :) | 22:18:29 |
| raitobezarius | Hydra dual-write is on my todo list | 22:35:36 |
| raitobezarius | And the tooling developed for the S3 GC might be reapplicable for the "hot cache" GC, even a simple LRU seems a good start (?), better policies can be deployed as things happen | 22:36:52 |
| raitobezarius | but I mostly agree with you delroth anyway | 22:36:56 |
| raitobezarius | Fastly dual-read seems something that can be done only by people having accesses after reading all the docs and ensuring the answer is not already there as far as I understand it | 22:37:35 |
| raitobezarius | (and I believe we concluded by 'yes' in the deduplication meetings, a fallback path can be implemented) | 22:37:53 |
| raitobezarius | (as with Hydra dual-sign ahem) | 22:38:51 |
| @delroth:delroth.net | In reply to @raitobezarius:matrix.org
Fastly dual-read seems something that can be done only by people having accesses after reading all the docs and ensuring the answer is not already there as far as I understand it it's not | 22:43:01 |
| @delroth:delroth.net | I'm the person who did the last major Fastly change for cache.nixos.org and I did 90% of the work with no access, fwiw | 22:43:38 |
| raitobezarius | well I imagine that you either read the documentation or did it on your own account | 22:43:53 |
| raitobezarius | which is what I meant by the end of my sentence | 22:44:00 |
| raitobezarius | or was there another trick I didn't think of? | 22:44:13 |
| raitobezarius | (but I think I agree with "this does not really really need Fastly access" from the get-go) | 22:44:37 |
| raitobezarius | https://github.com/NixOS/infra/issues/396 and https://github.com/NixOS/infra/issues/394 to keep track of dual write/read | 23:13:20 |
| 4 Mar 2024 |
 ajs124 | the hydra part could probably be done with a runcommand.
that way the rest of the configuration and how stuff gets copied to the main s3 wouldn't need to be modified. | 00:37:51 |
| @delroth:delroth.net | Is Hydra properly designed for having long running runcommands that can take minutes to run? I don't actually know how it's integrated with the queue runner, not even sure if it is | 02:45:03 |
| @delroth:delroth.net | Sorry, probably not the right place to have this discussion anyway | 02:45:58 |
| ajs124 | IME longer running runcommands aren't an issue, but I can skim the code to check if this is an obviously bad idea.
I'll comment my suggestion and findings on the issue later. | 09:31:53 |
| @delroth:delroth.net | Also we've completely disabled hydra-notify on h.n.o right now because each build completed notification fetches 250MB from the DB | 11:04:40 |
| @delroth:delroth.net | So that would have to be fixed first :) | 11:04:59 |
 edef | note that if we have dual reads, we can just have a separate service that does the copying to S3, if any | 14:50:11 |
