| 29 Feb 2024 |
 @admin:nixos.org | 👋 | 12:56:29 |
 Jonas Chevalier | this is getting very professional :) | 12:56:42 |
| @admin:nixos.org left the room. | 12:56:50 |
|  linj joined the room. | 12:58:10 |
|  mei 🌒& joined the room. | 13:28:31 |
|  hexa joined the room. | 13:43:29 |
|  Philip Taron (UTC-8) joined the room. | 17:16:37 |
| 1 Mar 2024 |
|  [0x4A6F] joined the room. | 19:37:08 |
| 3 Mar 2024 |
|  @janik0:matrix.org joined the room. | 13:17:14 |
|  Alyssa Ross joined the room. | 13:32:53 |
|  Julien joined the room. | 13:34:22 |
| 4 Mar 2024 |
|  nh2 joined the room. | 11:08:19 |
| nh2 | Regarding the current dedup effort, can it perform the dedup on the source (AWS) side, while storing the content-addressed objects on the remote side (like bup/``bupstash does), so that the outgoing traffic deduped (cheaper egress), but storage operations are done outside of AWS (e.g. free on self-hosted Hetzner)? | 12:46:52 |
| nh2 | * Regarding the current dedup effort, can it perform the dedup on the source (AWS) side, while storing the content-addressed objects on the remote side (like bup/``bupstash``` does), so that the outgoing traffic deduped (cheaper egress), but storage operations are done outside of AWS (e.g. free on self-hosted Hetzner)? | 12:47:10 |
| nh2 | * Regarding the current dedup effort, can it perform the dedup on the source (AWS) side, while storing the content-addressed objects on the remote side (like bup/bupstash does), so that the outgoing traffic deduped (cheaper egress), but storage operations are done outside of AWS (e.g. free on self-hosted Hetzner)? | 12:47:21 |
| nh2 | * Regarding the current dedup effort, can it perform the dedup on the source (AWS) side, while storing the content-addressed objects on the remote side (like bup/bupstash do), so that the outgoing traffic deduped (cheaper egress), but storage operations are done outside of AWS (e.g. free on self-hosted Hetzner)? | 12:47:32 |
 flokli | Yes | 12:57:06 |
| flokli | We could be reading through the NARs in the AWS bucket, decompress and CDC on a EC2 instance for example, then insert chunks into some hetzner object storage. however, xz decompression takes quite a big amount of CPU, so it might be quite slow, compared to snowballing out all, and running that part at Hetzner too. | 12:58:55 |
| nh2 | flokli: How much of the code to do that does already exist? | 13:00:48 |
| flokli | We have code essentially doing everything except actually persisting the chunks (only doing bookkeeping). That was used to test dedup ratios while we very chunking parameters | 13:02:01 |
| flokli | * We have code essentially doing everything except actually persisting the chunks (only doing bookkeeping). That was used to test dedup ratios while we vary chunking parameters | 13:02:38 |
| flokli | I'm also writing some code teaching tvix-castore how to use object storage. The internal model is the same, so it'd just be a matter of plugging some of this stuff together | 13:03:40 |
| nh2 | flokli: Is there a current write-up of the expected dedup ratios? | 13:04:50 |
| flokli | All that's on discourse, there's also a pad with our meeting notes linked somewhere there | 13:06:36 |
| flokli | (ah, channel topic here too) | 13:06:44 |
| nh2 | flokli: I have some troubles getting the latest numbers out of that pad.
In https://pad.lassul.us/nixos-cache-gc#Day-2023-11-07 it suggests dedup to 20% of original size, in https://pad.lassul.us/nixos-cache-gc#Day-2024-01-16 it says we got 30-40% better once we got enough data for the deduper, does that mean 14%? | 13:17:52 |
| flokli | We tested with "ingesting" 3 channel bumps, and the total size of chunks (which were zstd-compressed) was 20% less than the xz-compressed NAR files. That number is gonna go up the more things we ingest, as there might be more things that already exist. | 13:28:31 |
| flokli | We didn't play too much with chunking params, as popping the xz layer from the NARs is very very slow. took like 12h to read through one channel bump on the machine we had :-/ | 13:31:16 |
| nh2 | flokli: What kind of machine was it? From my memory XZ decompression is ~60 MB/s/core | 13:47:16 |
| nh2 | * flokli: What kind of machine was it? From my memory XZ decompression is ~60 MB/s/core (where 1 xz invocation does not scale beyond 1 core) | 13:48:09 |
