In reply to @edef1c:matrix.org
easy stats: over the last 24h we've served 2.1TiB from all our S3 buckets, uploaded 491G, in ~30M requests

Thanks!

So mean is 24 MB/s, 350 req/s.
That traffic will be very easy with 6 10 Gbit/s servers.
The IOPS should work, EC 6=4+2 should give 1500 reads IO/s, so we'd have 5x margin over the mean.

Probably a good amount of that can also be cached away from the HDDs as many people will be likely requesting latest nixos-* branches, and fewer people older, pinned branches.

In reply to @nh2:matrix.org

Thanks!

So mean is 24 MB/s, 350 req/s.
That traffic will be very easy with 6 10 Gbit/s servers.
The IOPS should work, EC 6=4+2 should give 1500 reads IO/s, so we'd have 5x margin over the mean.

Probably a good amount of that can also be cached away from the HDDs as many people will be likely requesting latest nixos-* branches, and fewer people older, pinned branches.

In reply to @nh2:matrix.org
edef: Can you query the number of files / the distribution/histogram of their sizes? A weakness of Ceph on HDDs is large amounts of small files.

In reply to @edef1c:matrix.org
from my larger sample, mean ~16MiB/s, median ~14MiB/s

In reply to @edef1c:matrix.org
note that i've given you zero information about I/O sizes so far

In reply to @edef1c:matrix.org
how much of those we're serving is a harder question, fastly's segmented caching makes it a bit weird to query that without a big painful join

I will likely not need serving numbers, the trouble with Ceph is usally storing the small files, because its maintenance operations (integrity "scrubs", "recovery" balancing on disk failure) are O(objects = files = seeks).

(For systems where the stored is bigger than what's served per month, which is the case here at 2TB/day.)

In reply to @edef1c:matrix.org
so you should be viewing the backend requests as (key, start offset, end offset) triples moreso than full fetches

In reply to @nh2:matrix.org
Will Fastly always chunk up the requests into small range requests, even if the user's Nix requests the whole NAR, or only if the end user requests a range?

In reply to @edef1c:matrix.org
i don't recall right now, sorry

Fastly segmented caching docs (https://docs.fastly.com/en/guides/segmented-caching#how-segmented-caching-works)

When an end user makes a Range: request for a resource with Segmented Caching enabled and a cache miss occurs (that is, at least part of the range is not cached), Fastly will make the appropriate Range: requests back to origin. Segmented Caching will then ensure only the specific portions of the resource that have been requested by the end user (along with rounding based on object size) will be cached rather than the entire resource. Partial cache hits will result in having the cached portion served from cache and the missing pieces fetched from origin. (Requests for an entire resource would be treated as a byte Range: request from 0 to end of resource.)

In reply to @edef1c:matrix.org
critical part being the final sentence

And the beginning of the paragraph:

When an end user makes a Range: request for a resource ...

This suggests "no range request by nix" => "no range request by Fastly to upstream"