| 1 Jun 2025 |
 Arian | I'm not saying we should push. I'm saying fastly should fetch from hydra instead of s3. And only fetch to s3 when hydra returns a 404 | 10:43:44 |
 emily | I guess the question is just whether Hydra can handle that load. and also how big its pipe to Fastly is? | 11:34:16 |
 edef | In reply to @arianvp:matrix.org
I'm not saying we should push. I'm saying fastly should fetch from hydra instead of s3. And only fetch to s3 when hydra returns a 404 we could alternatively finish the Tigris story / move more towards s3 as pure archival storage | 11:35:31 |
| edef | both tigris and r2 are free egress, the norm is changing gradually | 11:36:06 |
| Arian | In reply to @emilazy:matrix.org
I guess the question is just whether Hydra can handle that load. and also how big its pipe to Fastly is? It is already handling that load though? It's uploading to S3 after all. | 11:41:11 |
 toonn | That load would be almost doubling, no? | 11:48:45 |
| edef | no, we could run the pipeline through fastly, if we wanted to | 11:49:13 |
| edef | we can fetch into s3 on the aws side, pulling from fastly | 11:50:12 |
| edef | whether that is wise is another question, but we can definitely do it | 11:50:27 |
| edef | mostly it is isomorphic to having hydra have a more-local / free-egress storage service | 11:51:49 |
| edef | and using s3 mostly as archival storage | 11:52:05 |
| toonn | That's certainly better than Hydra pushing to S3 and Fastly pulling from it preferentially but won't most of the Fastly cache still be clobbered by the S3 requests? So in the end Fastly would fetch many things from Hydra twice still? | 12:29:43 |
 flokli | We can also do the push through thing. So instead of hydra uploading to fastly or s3 directly, it'd send to a http daemon that uploads to both places | 13:47:44 |
| flokli | It gives nicer control over the import path | 13:48:01 |
| edef | yes, these are all essentially identical in terms of dataflow | 13:48:06 |
| flokli | * It gives nicer control over the upload path | 13:48:12 |
| edef | * yes, these are all essentially identical in terms of bulk data flow | 13:48:20 |
| edef | or well, we would be spending twice the hetzner bandwidth if we go to both | 13:48:57 |
| edef | mostly it depends on how much annoying orchestration we want to have | 13:49:16 |
| edef | * mostly it depends on how much annoying orchestration and consistency trickery we want to have | 13:49:29 |
| flokli | I assumed network bandwidth egressing from Hetzner is not the bottleneck | 13:51:01 |
 hexa | it is not | 13:51:12 |
| edef | it's not, no | 13:51:15 |
| edef | so like, for the most part you can do anything to the NAR datapath | 13:52:06 |
| edef | as long as narinfo 404 is fast | 13:52:16 |
|  BMG joined the room. | 15:29:29 |
| Arian | In reply to @flokli:matrix.org
We can also do the push through thing. So instead of hydra uploading to fastly or s3 directly, it'd send to a http daemon that uploads to both places I'm suggesting something way simpler.
We just have nix-serve running on hydra and serve its nix store.
We configure fastly to first fetch Hydra and on 404 fetch s3 | 17:41:24 |
| Arian | Requires no big changes on hydra side | 17:41:36 |
| edef | the narinfo 404 path is sacred, we cause a lot of user suffering if it isn't fast | 17:42:18 |
| Arian | But it might be too simplistic. | 17:42:18 |
