| 3 Mar 2024 |
 raitobezarius | which is what I meant by the end of my sentence | 22:44:00 |
| raitobezarius | or was there another trick I didn't think of? | 22:44:13 |
| raitobezarius | (but I think I agree with "this does not really really need Fastly access" from the get-go) | 22:44:37 |
| raitobezarius | https://github.com/NixOS/infra/issues/396 and https://github.com/NixOS/infra/issues/394 to keep track of dual write/read | 23:13:20 |
| 4 Mar 2024 |
 ajs124 | the hydra part could probably be done with a runcommand.
that way the rest of the configuration and how stuff gets copied to the main s3 wouldn't need to be modified. | 00:37:51 |
 delroth | Is Hydra properly designed for having long running runcommands that can take minutes to run? I don't actually know how it's integrated with the queue runner, not even sure if it is | 02:45:03 |
| delroth | Sorry, probably not the right place to have this discussion anyway | 02:45:58 |
| ajs124 | IME longer running runcommands aren't an issue, but I can skim the code to check if this is an obviously bad idea.
I'll comment my suggestion and findings on the issue later. | 09:31:53 |
| delroth | Also we've completely disabled hydra-notify on h.n.o right now because each build completed notification fetches 250MB from the DB | 11:04:40 |
| delroth | So that would have to be fixed first :) | 11:04:59 |
 edef | note that if we have dual reads, we can just have a separate service that does the copying to S3, if any | 14:50:11 |
| edef | the perf requirements on the storage backends are really loose, except for serving 404s | 14:50:34 |
| edef | * the perf requirements on the storage backends are really loose, except for serving narinfo 404s | 14:50:55 |
| edef | but narinfo keys are only 5G of stuff, we can serve 404s basically however we want | 14:51:22 |
| edef | any actual serving ends up being from fastly and too heavily cached for the backend's perf to matter | 14:52:17 |
 Jonas Chevalier | nh2: do you want to join the infra meeting on Thusday 18:00 GMT+1 and hash this out with us? | 14:55:08 |
| raitobezarius | Isn't delroth going to be off? | 14:55:46 |
| raitobezarius | I think it's good to have delroth on those discussions | 14:55:57 |
| Jonas Chevalier | it's fine, we already discussed this | 14:57:12 |
| Jonas Chevalier | we have the overall ideas but what's missing is to map some of the unknowns, like delroth said; can we make this sustainable, what a migration path looks like, ... | 15:00:45 |
| Jonas Chevalier | we might be able to port other things than the cache first | 15:01:56 |
| edef | so like, my biggest proposal wrt the cache GC is that we aggregate the "deleted" data into Glacier Deep Archive, as large objects | 22:12:32 |
| edef | that locks us in for 6 months, but if there are no other takers, i'll put down the $3k to buy myself 6 months of development time for an exit strategy from AWS for that data | 22:13:19 |
| edef | it should dedupe quite well, but my biggest issue is simply time pressure | 22:14:24 |
| edef | i don't really intend don't intend to lose a single byte of that historical data but the stress from trying to do all of this fast is weighing on me | 22:16:33 |
| edef | the narinfo dataset is archived in several places now, that part we have covered | 22:17:52 |
| raitobezarius | In reply to @edef1c:matrix.org
that locks us in for 6 months, but if there are no other takers, i'll put down the $3k to buy myself 6 months of development time for an exit strategy from AWS for that data we collected enough money to put the 3K as part of the "binary cache niceties" budget | 22:18:04 |
| raitobezarius | fwiw | 22:18:09 |
| edef | sure, that works for me, $3k is def still a meaningful cost for me | 22:18:32 |
| edef | i just know that history would not judge me kindly if i let this data go to /dev/null | 22:19:13 |
