WHERE NOT key REGEXP '^[0123456789abcdfghijklmnpqrsvwxyz]{32}\.narinfo$'
  AND NOT key REGEXP '^[0123456789abcdfghijklmnpqrsvwxyz]{32}\.ls(\.xz)?$'
  AND NOT key REGEXP '^[0123456789abcdfghijklmnpqrsvwxyz]{32}-[a-zA-Z0-9+\-_?=][a-zA-Z0-9+\-_?=.]*\.ls$'
  AND NOT key REGEXP '^nar/[0123456789abcdfghijklmnpqrsvwxyz]{52}\.nar\.(bz2|xz)$'
  AND NOT key REGEXP '^log/[0123456789abcdfghijklmnpqrsvwxyz]{32}-[a-zA-Z0-9+\-_?=][a-zA-Z0-9+\-_?=.]*\.drv$'
  AND NOT key REGEXP '^debuginfo/[0-9a-f]{40}$'
  AND NOT key REGEXP '^debuginfo/[0-9a-f]{16}$'
  AND NOT key IN ('.well-known/pki-validation/gsdv.txt', 'nix-cache-info', 'index.html', 'binary-cache/', 'error-pages/403', 'error-pages/404')

that is, the following query yields 7400044328 bytes/min ≈ 1 Gbit/s

SELECT max(bytes_per_minute)
FROM
(
    SELECT
        toStartOfMinute(fromUnixTimestamp(timestamp)) AS minute,
        SUM(bytes_sent) AS bytes_per_minute
    FROM s3('s3://nix-archeologist/nix-cache-bucket-logs/*.parquet', Parquet, 'key String, http_status Int64, bytes_sent Int64, user_agent String, operation String, requester String, timestamp UInt32')
    WHERE startsWith(key, 'nar/') AND (requester = 'arn:aws:iam::080433136561:user/fastly-cache-access') AND (http_status IN (200, 206))
    GROUP BY minute
)

In reply to @edef1c:matrix.org
easy stats: over the last 24h we've served 2.1TiB from all our S3 buckets, uploaded 491G, in ~30M requests

Thanks!

So mean is 24 MB/s, 350 req/s.
That traffic will be very easy with 6 10 Gbit/s servers.
The IOPS should work, EC 6=4+2 should give 1500 reads IO/s, so we'd have 5x margin over the mean.

Probably a good amount of that can also be cached away from the HDDs as many people will be likely requesting latest nixos-* branches, and fewer people older, pinned branches.

In reply to @nh2:matrix.org

Thanks!

So mean is 24 MB/s, 350 req/s.
That traffic will be very easy with 6 10 Gbit/s servers.
The IOPS should work, EC 6=4+2 should give 1500 reads IO/s, so we'd have 5x margin over the mean.

Probably a good amount of that can also be cached away from the HDDs as many people will be likely requesting latest nixos-* branches, and fewer people older, pinned branches.

In reply to @nh2:matrix.org
edef: Can you query the number of files / the distribution/histogram of their sizes? A weakness of Ceph on HDDs is large amounts of small files.

In reply to @edef1c:matrix.org
from my larger sample, mean ~16MiB/s, median ~14MiB/s

In reply to @edef1c:matrix.org
note that i've given you zero information about I/O sizes so far

In reply to @edef1c:matrix.org
how much of those we're serving is a harder question, fastly's segmented caching makes it a bit weird to query that without a big painful join

I will likely not need serving numbers, the trouble with Ceph is usally storing the small files, because its maintenance operations (integrity "scrubs", "recovery" balancing on disk failure) are O(objects = files = seeks).

(For systems where the stored is bigger than what's served per month, which is the case here at 2TB/day.)