!CcTBuBritXGywOEGWJ:matrix.org

NixOS Binary Cache Self-Hosting

172 Members
About how to host a very large-scale binary cache and more58 Servers

Load older messages


SenderMessageTime
7 Mar 2024
@whentze:matrix.orgWanja Hentzeeh, for a loooot of NixOS users, the cache being down means nixos-rebuild just becomes a desk warming app00:19:53
@edef1c:matrix.orgedefyeah00:20:03
@edef1c:matrix.orgedeffrom the cache logs, i know we have a lot of users in like, low-GDP countries you've never heard of before00:20:30
@whentze:matrix.orgWanja Hentzewe can actually build our closures on our CI infra but I'm sure many can't00:20:36
@edef1c:matrix.orgedefi don't think they would appreciate the desk warming since it is coming out of their power bill00:20:46
@whentze:matrix.orgWanja Hentzeand in my hobby usage I definitely can't00:20:55
@edef1c:matrix.orgedefnor do they likely have the RAM to actually build most things00:21:11
@raitobezarius:matrix.orgraitobezarius
In reply to @edef1c:matrix.org
from the cache logs, i know we have a lot of users in like, low-GDP countries you've never heard of before
i wonder if low GDP countries can download fast enough some of our stuff though
00:21:31
@raitobezarius:matrix.orgraitobezarius and if they don't end up passing --no-substituters or something 00:21:42
@edef1c:matrix.orgedefi'm working from cache logs, so they do seem to hit it00:22:01
@edef1c:matrix.orgedefor they wouldn't show on my radar at all00:22:05
@raitobezarius:matrix.orgraitobezariusah right00:22:05
@edef1c:matrix.orgedefbandwidth will hopefully improve in the long run if we can get deduping to work well and carry all the way through to the frontend protocol00:22:55
@whentze:matrix.orgWanja Hentzeanyone familiar with the micromirror project? https://blog.thelifeofkenneth.com/2023/05/building-micro-mirror-free-software-cdn.html?m=100:25:25
@whentze:matrix.orgWanja Hentzethey're just chucking shitty tiny boxes everywhere to hit the hottest cache entries really fast (because closes to users)00:26:01
@whentze:matrix.orgWanja Hentze* they're just chucking shitty tiny boxes everywhere to hit the hottest cache entries really fast (because closer to users)00:26:15
@edef1c:matrix.orgedefwe have Fastly covering that and i have some other stuff i could call00:26:33
@edef1c:matrix.orgedef * we have Fastly covering that and i have some other stuff i could call in00:26:37
@edef1c:matrix.orgedefbroadly CDN is not the biggest thing to worry about right this second00:27:04
@whentze:matrix.orgWanja Hentzeoh, so the plan is not to self host the CDN00:27:09
@whentze:matrix.orgWanja Hentzethat makes it much easier indeed00:27:15
@raitobezarius:matrix.orgraitobezarius
In reply to @whentze:matrix.org
oh, so the plan is not to self host the CDN
in some very distant long term super future where Fastly may drop us
00:27:22
@edef1c:matrix.orgedef
In reply to @whentze:matrix.org
oh, so the plan is not to self host the CDN
yeah, not in the short term anyway
00:27:29
@edef1c:matrix.orgedefi'd like to have a more worked out story for "what if Fastly dies / quits on us" but i'm broadly satisfied with our risk exposures there00:28:01
@delroth:delroth.netdelroth
In reply to @linus:schreibt.jetzt
we also have the advantage of the cache not being super critical for nix/nixos to be usable
I think it's part of the prerequisite work for self hosting to actually determine answers to those questions (at least to some level of consensus). My personal opinion is different from yours there - I think that (recent, not historical) cache unavailability is very visible and very impactful for nixpkgs users.
00:28:28
@edef1c:matrix.orgedefdoesn't mean we can't diversify if it's easy or cheap00:28:33
@edef1c:matrix.orgedef
In reply to @delroth:delroth.net
I think it's part of the prerequisite work for self hosting to actually determine answers to those questions (at least to some level of consensus). My personal opinion is different from yours there - I think that (recent, not historical) cache unavailability is very visible and very impactful for nixpkgs users.
i mean, we can always turn it off for a bit if we want to find out
00:28:51
@whentze:matrix.orgWanja Hentzethe ol' yank 'n' yell00:29:51
@edef1c:matrix.orgedefin part by how reliable it has been, we've created this dependence00:30:41
@edef1c:matrix.orgedeflike, if we exceed our SLOs by far, we ought to blink it00:31:26

Show newer messages


Back to Room ListRoom Version: 10