| 21 Aug 2023 |
 @elvishjerricco:matrix.org | plus I think optane has 512B sectors, so ashift=9 on that vdev 😎 | 09:05:39 |
 @linus:schreibt.jetzt | but are you going to use raidz2 on your special too? | 09:05:39 |
| @elvishjerricco:matrix.org | you can't use raidz on special I'm pretty sure | 09:05:53 |
| @elvishjerricco:matrix.org | ashift=9 on the special just means that small things get even smaller | 09:06:41 |
| @linus:schreibt.jetzt | In reply to @elvishjerricco:matrix.org
you can't use raidz on special I'm pretty sure ah yeah | 09:07:03 |
| @elvishjerricco:matrix.org | I've only got two of these optanes in here, so my redundancy level is sort of different. But it's optane, so it should resilver fast in the event of a failure, and they should be durable as hell. So it's not too big a worry | 09:08:00 |
| @linus:schreibt.jetzt | yeah, and losing a binary cache isn't the end of the world I think? | 09:08:31 |
| @linus:schreibt.jetzt | and you'd have backups of any more important stuff, right? ;) | 09:08:43 |
| @elvishjerricco:matrix.org | right; granted there's other stuff I like on this pool but none of it matters | 09:08:45 |
| @elvishjerricco:matrix.org | and yes :P | 09:08:53 |
| @elvishjerricco:matrix.org | other machine has a single very big hard drive as a send/recv target | 09:09:28 |
| @linus:schreibt.jetzt | In reply to @julienmalka:matrix.org
On my own attic deployment I get 0.17052771681498935 you can also replace file_size with chunk_size to see the ratio for dedup-only (without compression) | 09:09:47 |
| @linus:schreibt.jetzt | which is like 47% for me | 09:09:55 |
| @linus:schreibt.jetzt | * which is like 45% for me | 09:10:05 |
 Julien | In reply to @linus:schreibt.jetzt
which is like 45% for me Yeah 43% | 09:11:33 |
| Julien | So this is still a big time storage economy | 09:12:15 |
| @elvishjerricco:matrix.org | Linux Hackerman: Oh wow yea. Tested out copying a big closure from the cache and the "querying path" stuff was very noticeably stop-and-go and took a good amount of time.
I've now added optane special metadata and migrated the data with a send/recv to and from the same pool. I used a completely different closure to make sure it wasn't just remembering the cache hit, and that querying part is effectively instant now
| 11:24:55 |
| @linus:schreibt.jetzt | \o/ | 11:25:09 |
| @elvishjerricco:matrix.org | optane is kind of insane | 11:25:27 |
| @elvishjerricco:matrix.org | there was a huge influx of supply like 6 months ago or something when intel announced they were killing the division, so I managed to pick up four 110GB nvme sticks at a quarter their normal price | 11:26:02 |
| @linus:schreibt.jetzt | nice | 11:26:15 |
| 22 Aug 2023 |
| @elvishjerricco:matrix.org | whoa; just did a full system update with the "new" cache storage, and while optane made the querying instant, the download bandwidth was way reduced. zpool iostat 1 showed a pretty steady read bandwidth of ~80MB/s but I wasn't getting anywhere near that over the wire; whereas on the SSD I saturated the gigabit easily | 05:30:57 |
| @elvishjerricco:matrix.org | (also this pool can easily do sequential reads upwards of 500MB/s) | 05:32:30 |
| @linus:schreibt.jetzt | huh | 06:38:37 |
| @linus:schreibt.jetzt | did you rewrite the nars as well? | 06:38:41 |
| @linus:schreibt.jetzt | I could imagine that happening if you're downloading a lot of small nars, but wouldn't expect it for big ones | 06:38:58 |
| @linus:schreibt.jetzt | * I could imagine that happening if you're downloading a lot of small nars and those nars are on HDD, but wouldn't expect it for big ones | 06:39:20 |
 Zhaofeng Li | In reply to @elvishjerricco:matrix.org
I didn't realize accessing narinfos was such a burden (finally trying to catch up with stuff)
narinfos aren't actually individual small files, the server is doing a database query | 13:46:12 |
| 23 Aug 2023 |
|  Sofi joined the room. | 00:01:06 |
| @elvishjerricco:matrix.org | In reply to @linus:schreibt.jetzt
did you rewrite the nars as well? yep. Did a send/recv of the whole dataset. Maybe I needed to make the small blocks property a little bigger | 04:07:56 |
