| 7 Mar 2024 |
 raitobezarius | what is the process / is it fine / is it bad to download the last 2-3 releases of each channels directly? | 00:34:36 |
| raitobezarius | over Fastly | 00:34:38 |
| raitobezarius | or does it thrash the CDN to your opinion? | 00:34:44 |
| raitobezarius | I don't have the data | 00:34:45 |
| raitobezarius | should I do S3 Requester Pays stuff for that | 00:35:04 |
 edef | Fastly is fairly okay about scans, but it would be nice if you hit the bucket with requester pays | 00:35:08 |
| raitobezarius | okie | 00:35:29 |
| raitobezarius | i might bother you later on with some queries | 00:35:33 |
| raitobezarius | (or you should teach me a bit your clickhouse workflow and i will mimic it so i become autonomous) | 00:36:27 |
| edef | if you want a lot of stuff, we can plausibly pull and dedupe it for you EC2-side | 00:36:27 |
| raitobezarius | i'm basically working on a project to enable a small 2x10Gbps appliance to serve offline "fresh" nixpkgs/nixos | 00:36:52 |
| raitobezarius | and in that sense, i need to do regular syncs with last N releases of nixos | 00:37:47 |
| edef | In reply to @whentze:matrix.org
so the thing to self host would be large, robust, highly available but not necessarily screaming fast (object?) storage? to elaborate on this more: we basically have three tiers. frontend cache, semi-hot fast-enough storage, cold storage / DR | 00:37:51 |
| raitobezarius | (actually i mostly need latest release) | 00:37:53 |
| edef | hmm, okay | 00:38:01 |
| raitobezarius | but ideally stable and unstable | 00:38:15 |
| edef | obviously requester pays means it's none of my problem in that capacity | 00:38:19 |
| edef | but deduped cache replication might be positive-sum here | 00:38:38 |
| raitobezarius | i'm okay funding deduped cache replication via my own requester pays stuff to avoid putting any churn on the foundation | 00:38:58 |
| edef | In reply to @edef1c:matrix.org
to elaborate on this more: we basically have three tiers. frontend cache, semi-hot fast-enough storage, cold storage / DR one of the other considerations is like, the ingestion path leads straight into the AWS prison | 00:39:03 |
| edef | In reply to @edef1c:matrix.org
to elaborate on this more: we basically have three tiers. frontend cache, semi-hot fast-enough storage, cold storage / DR * one of the other considerations is like, the ingestion path leads straight into the AWS prison right now | 00:39:08 |
| raitobezarius | but my stuff is mostly a tangent and i will go to bed | 00:39:24 |
| edef | i'd much rather have that piped into somewhere cheap and from there to S3, or replicated into two places | 00:39:40 |
| edef | there just isn't a separate ingestion mechanism right now, the build farm hits S3 directly | 00:39:53 |
| raitobezarius | to be discussed next meeting I guess | 00:40:00 |
| edef | i'm not even sure it fetches from Fastly | 00:40:02 |
| raitobezarius | I need to hammer a bit more the software | 00:40:04 |
| raitobezarius | to make this more tangible | 00:40:07 |
 Zhaofeng Li | <offtopic>Ah, Requester Pays... I always wondered why AWS/Google don't provide an easy way for non-technical people to access those buckets (i.e., enter credit card and bucket name, get web access)</offtopic> | 01:02:24 |
| edef | hmm, that seems buildable | 01:02:53 |
