| 21 Aug 2023 |
 @linus:schreibt.jetzt | maybe keep the nars on the SSD though | 08:40:26 |
| @linus:schreibt.jetzt | err | 08:40:31 |
| @linus:schreibt.jetzt | narinfos* | 08:40:32 |
 @elvishjerricco:matrix.org | oh | 08:40:36 |
| @elvishjerricco:matrix.org | that's a good idea | 08:40:39 |
| @elvishjerricco:matrix.org | whoops | 08:40:40 |
| @elvishjerricco:matrix.org | too late :P | 08:40:44 |
| @elvishjerricco:matrix.org | might change that later | 08:40:56 |
| @linus:schreibt.jetzt | from my experience, an all-HDD binary cache is pretty painful | 08:41:48 |
| @elvishjerricco:matrix.org | I didn't realize accessing narinfos was such a burden | 08:42:32 |
| @linus:schreibt.jetzt | but if you have a special device on your zpool and set special_small_blocks you can get your narinfos on the SSD within the pool | 08:42:52 |
| @linus:schreibt.jetzt | with the bonus of getting small nars on the SSD too | 08:43:01 |
| @linus:schreibt.jetzt | In reply to @elvishjerricco:matrix.org
I didn't realize accessing narinfos was such a burden lots of small files access, lots of very random access. Worst performance case for HDDs ^^ | 08:43:38 |
| @elvishjerricco:matrix.org | I do wish I had a special device in this pool. And I have the hardware for it (yay optane). But I've been planning on rebuilding the pool for a while and planned on adding them then | 08:44:02 |
| @elvishjerricco:matrix.org | In reply to @linus:schreibt.jetzt
lots of small files access, lots of very random access. Worst performance case for HDDs ^^ Yea I just didn't realize narinfos would be a significant amount of the IOPS necessary to serve a binary cache | 08:44:59 |
| @linus:schreibt.jetzt | In reply to @elvishjerricco:matrix.org
I do wish I had a special device in this pool. And I have the hardware for it (yay optane). But I've been planning on rebuilding the pool for a while and planned on adding them then why only then? ^^ | 08:45:26 |
| @elvishjerricco:matrix.org | Because I didn't know I'd have a legit use for it this very night :P | 08:45:44 |
| @linus:schreibt.jetzt | :D | 08:45:49 |
| @linus:schreibt.jetzt | though briefly back to the topic of attic's deduplication: I'm using the suggested chunking parameters, and they result in uploads being fairly slow (the bottleneck appears to be CPU-bound stuff in the postgres database) | 08:47:16 |
| @elvishjerricco:matrix.org | I'm not hugely worried about anything like that. More worried about download speed | 08:48:05 |
| @elvishjerricco:matrix.org | Using this on my LAN | 08:48:10 |
| @linus:schreibt.jetzt | download speed is fine, reading the database is fast and reassembling the nars from chunks is trivial | 08:48:37 |
| @elvishjerricco:matrix.org | though, the single threaded perf of this CPU aint great | 08:48:39 |
 Julien | Do we have measures of the total deduplication of an attic cache ? | 08:48:57 |
| @elvishjerricco:matrix.org | In reply to @linus:schreibt.jetzt
why only then? ^^ I just remembered... I've already got the optane in the machine... I could just add it now and rewrite the narinfo files... | 08:49:53 |
|  Christina Sørensen joined the room. | 08:50:44 |
| @linus:schreibt.jetzt | In reply to @julienmalka:matrix.org
Do we have measures of the total deduplication of an attic cache ? attic doesn't currently have a built-in command for it but it should be pretty easy to get from the db | 08:51:14 |
| @linus:schreibt.jetzt | attic=# select sum(file_size) / (select sum(nar_size) from nar) from chunk;
?column?
------------------------
0.20098691256561418637
(1 row)
| 08:52:57 |
| @linus:schreibt.jetzt | so I guess I'm using 20% as much storage for nars as I would be without dedup | 08:53:15 |
| @linus:schreibt.jetzt | *and compression | 08:53:30 |
