| 7 Mar 2024 |
 raitobezarius | (or you should teach me a bit your clickhouse workflow and i will mimic it so i become autonomous) | 00:36:27 |
 edef | if you want a lot of stuff, we can plausibly pull and dedupe it for you EC2-side | 00:36:27 |
| raitobezarius | i'm basically working on a project to enable a small 2x10Gbps appliance to serve offline "fresh" nixpkgs/nixos | 00:36:52 |
| raitobezarius | and in that sense, i need to do regular syncs with last N releases of nixos | 00:37:47 |
| edef | In reply to @whentze:matrix.org
so the thing to self host would be large, robust, highly available but not necessarily screaming fast (object?) storage? to elaborate on this more: we basically have three tiers. frontend cache, semi-hot fast-enough storage, cold storage / DR | 00:37:51 |
| raitobezarius | (actually i mostly need latest release) | 00:37:53 |
| edef | hmm, okay | 00:38:01 |
| raitobezarius | but ideally stable and unstable | 00:38:15 |
| edef | obviously requester pays means it's none of my problem in that capacity | 00:38:19 |
| edef | but deduped cache replication might be positive-sum here | 00:38:38 |
| raitobezarius | i'm okay funding deduped cache replication via my own requester pays stuff to avoid putting any churn on the foundation | 00:38:58 |
| edef | In reply to @edef1c:matrix.org
to elaborate on this more: we basically have three tiers. frontend cache, semi-hot fast-enough storage, cold storage / DR one of the other considerations is like, the ingestion path leads straight into the AWS prison | 00:39:03 |
| edef | In reply to @edef1c:matrix.org
to elaborate on this more: we basically have three tiers. frontend cache, semi-hot fast-enough storage, cold storage / DR * one of the other considerations is like, the ingestion path leads straight into the AWS prison right now | 00:39:08 |
| raitobezarius | but my stuff is mostly a tangent and i will go to bed | 00:39:24 |
| edef | i'd much rather have that piped into somewhere cheap and from there to S3, or replicated into two places | 00:39:40 |
| edef | there just isn't a separate ingestion mechanism right now, the build farm hits S3 directly | 00:39:53 |
| raitobezarius | to be discussed next meeting I guess | 00:40:00 |
| edef | i'm not even sure it fetches from Fastly | 00:40:02 |
| raitobezarius | I need to hammer a bit more the software | 00:40:04 |
| raitobezarius | to make this more tangible | 00:40:07 |
 Zhaofeng Li | <offtopic>Ah, Requester Pays... I always wondered why AWS/Google don't provide an easy way for non-technical people to access those buckets (i.e., enter credit card and bucket name, get web access)</offtopic> | 01:02:24 |
| edef | hmm, that seems buildable | 01:02:53 |
| edef | i think their ToS might have some stuff about reselling but i'm not sure how much that applies here | 01:03:32 |
| edef | * i think their ToS might have some stuff about reselling their services too directly but i'm not sure how much that applies here | 01:03:48 |
| edef | but you can probably go forth and be the change in the world you want to see | 01:04:11 |
| raitobezarius | aka build a startup on the top of that | 01:04:25 |
| Zhaofeng Li | It would be nice if such an UI can avoid being in the middle of the payment process | 01:05:03 |
| Zhaofeng Li | like research teams can self-host the UI and not have to worry about fees, tax implications etc | 01:05:36 |
| edef | you can totally sign up for an AWS account and use the bucket explorer UI, i think? | 01:05:39 |
| Zhaofeng Li | * like research teams (dataset publishers) can self-host the UI and not have to worry about fees, tax implications etc | 01:05:55 |
