!CcTBuBritXGywOEGWJ:matrix.org

NixOS Binary Cache Self-Hosting

170 Members
About how to host a very large-scale binary cache and more59 Servers

Load older messages


SenderMessageTime
27 Jan 2024
@lehmanator:gnulinux.clubSam Lehman removed their profile picture.16:55:25
@lehmanator:gnulinux.clubSam Lehman removed their display name Sam Lehman.17:02:41
@lehmanator:gnulinux.clubSam Lehman left the room.17:05:45
29 Jan 2024
@lehmanator:tchncs.deSam Lehman joined the room.11:04:17
@lehmanator:tchncs.deSam Lehman set a profile picture.11:06:03
@flandweber:envs.netFinn Landweber joined the room.15:01:45
@flandweber:envs.netFinn Landweber changed their display name from flandweber to Finn Landweber.18:20:07
31 Jan 2024
@federicodschonborn:matrix.orgFederico Damián Schonborn changed their profile picture.03:36:27
@federicodschonborn:matrix.orgFederico Damián Schonborn changed their profile picture.06:22:05
@zimbatm:numtide.comJonas Chevalierhttps://discourse.nixos.org/t/upcoming-garbage-collection-for-cache-nixos-org/3907813:50:00
12 Feb 2024
@zimbatm:numtide.comJonas ChevalierWhy Tarnsap doesn't use Glacier: https://www.daemonology.net/blog/2012-09-04-why-tarsnap-doesnt-use-glacier.html15:59:47
@raitobezarius:matrix.orgraitobezariusyeah edef is in contact with Colin16:00:41
@zimbatm:numtide.comJonas Chevalierthe TL;DR is that if you do chunk deduplication, then recent objects might have chunks stored in Glacier. And Glacier retrieval is super expensive.16:09:58
@hexa:lossy.networkhexaare the goals listed in the attic readme all realized?22:33:33
@hexa:lossy.networkhexaor are they still goals?22:33:42
@zhaofeng:zhaofeng.liZhaofeng LiYes, they are realized. One thing is that scaling can be bottlenecked by the underlying database22:35:53
13 Feb 2024
@artemist:mildlyfunctional.gayartemistOn the topic of attic, are there places people suggest hosting the backing objects? I've been using b2 but find it very unreliable and slow00:11:32
@zhaofeng:zhaofeng.liZhaofeng Li I'm using backblaze r2 which is more expensive storage-wise but charges nothing for outbound 00:12:46
@zhaofeng:zhaofeng.liZhaofeng Li * I'm using cloudflare r2 which is more expensive storage-wise but charges nothing for outbound 00:13:11
@zhaofeng:zhaofeng.liZhaofeng Li (s/backblaze/cloudflare duh) 00:13:26
@artemist:mildlyfunctional.gayartemist
In reply to @zhaofeng:zhaofeng.li
I'm using cloudflare r2 which is more expensive storage-wise but charges nothing for outbound
Thanks, I'll look at that soon. For now I'm just moving stuff back to the local disk on my VPS
00:17:01
@edef1c:matrix.orgedef joined the room.15:04:29
@edef1c:matrix.orgedef
In reply to @zimbatm:numtide.com
the TL;DR is that if you do chunk deduplication, then recent objects might have chunks stored in Glacier. And Glacier retrieval is super expensive.
yes, the answer is to just dupe the chunks in that case
15:04:56
@edef1c:matrix.orgedefthe goal isn't "store every chunk exactly once", that's not an inherent requirement15:09:57
@flokli:matrix.orgflokliWell, and it only matters if you need to navigate the AWS pricing landscape15:10:40
@flokli:matrix.orgflokli If we self-host the binary cache, we probably won't add our own billing department 15:11:04
@edef1c:matrix.orgedef
In reply to @artemist:mildlyfunctional.gay
On the topic of attic, are there places people suggest hosting the backing objects? I've been using b2 but find it very unreliable and slow
yeah backblaze is absolute clown shoes, it just be like that
15:11:20
@edef1c:matrix.orgedef
In reply to @flokli:matrix.org
If we self-host the binary cache, we probably won't add our own billing department
i'm afraid disk iops are still finite resources no matter how you slice it
15:11:44
@julienmalka:matrix.orgJulien
In reply to @flokli:matrix.org
If we self-host the binary cache, we probably won't add our own billing department
Is this on the table you think ?
15:12:36
@flokli:matrix.orgflokliAck, but I don't think the disk iops will be the problem, the actual traffic we egress from the bucket is fairly slim15:12:44

Show newer messages


Back to Room ListRoom Version: 10