!CcTBuBritXGywOEGWJ:matrix.org

NixOS Binary Cache Self-Hosting

158 Members
About how to host a very large-scale binary cache and more54 Servers

Load older messages


SenderMessageTime
7 Mar 2024
@edef1c:matrix.orgedefthe people you retain are the ones for whom the incentives work00:02:32
@edef1c:matrix.orgedefthis scene has bled people pretty hard over the ~8y i've been part of the show00:03:10
@adam:robins.wtf@adam:robins.wtfI do think we should expect to pay someone for an ongoing number of hours. While it may not be necessary to care and feed a system every day, I wouldn't underestimate the potential problems or just unforeseen maintenance of a storage cluster.00:03:13
@edef1c:matrix.orgedefpart of the thing here is, you can prevent a lot of emergencies by doing consistent work to make them unlikely and handleable00:04:27
@edef1c:matrix.orgedefspes non consilium est; hope is not a strategy00:04:49
@edef1c:matrix.orgedefconsistently building reliable systems and doing reliability work instead of responding to incidents is how we make this good00:05:19
@edef1c:matrix.orgedefand this buys us various kinds of budget to do the improving, it compounds well00:05:40
@edef1c:matrix.orgedeflike, what i want from the incentive alignment is avoidance of "no, we won't pay you to spend time hacking on making this system more stable, but we will pay you to respond to the emergencies this creates"00:06:25
@adam:robins.wtf@adam:robins.wtfmakes sense to me00:07:20
@whentze:matrix.orgWanja Hentze
In reply to @edef1c:matrix.org
part of the thing here is, you can prevent a lot of emergencies by doing consistent work to make them unlikely and handleable
yes. I'm currently on-call. The last time I was paged was months ago, and it was a false positive. the time before that was years ago.
00:07:30
@adam:robins.wtf@adam:robins.wtfthat's the dream rotation00:07:45
@raitobezarius:matrix.orgraitobezarius
In reply to @whentze:matrix.org
yes. I'm currently on-call. The last time I was paged was months ago, and it was a false positive. the time before that was years ago.
do we need to make things spicier in nixos
00:07:51
@raitobezarius:matrix.orgraitobezariussorry00:07:53
@edef1c:matrix.orgedef
In reply to @whentze:matrix.org
yes. I'm currently on-call. The last time I was paged was months ago, and it was a false positive. the time before that was years ago.
and thus it would be a reasonable bet for me to take that you work in an environment where those incentives are well-aligned
00:08:01
@adam:robins.wtf@adam:robins.wtfi've definitely had on-call rotations that were much more painful than a couple times a year :)00:08:09
@whentze:matrix.orgWanja Hentzeif you do a good job on the infra and have reasonable requirements (this one is important), on-call can be relatively okay00:08:19
@whentze:matrix.orgWanja Hentzethat being said I would currently not join a 2nd rotation because I simply need time off for real once in a while00:11:49
@whentze:matrix.orgWanja HentzeI can help with infra work though if you want me to beat a NixOS over the head or replace a hard drive or whatever00:13:21
@linus:schreibt.jetzt@linus:schreibt.jetztYeah I'm not sure "on-call" makes that much sense here anyway? I'd consider it to be a best-effort thing, and I expect that if stuff catches fire people will scramble to fix it, but nobody needs to be paged out of whatever they're doing for it00:15:25
@whentze:matrix.orgWanja Hentzeidk, crates.io has a rotation and I consider that less essential00:16:07
@linus:schreibt.jetzt@linus:schreibt.jetzt we also have the advantage of the cache not being super critical for nix/nixos to be usable 00:16:16
@linus:schreibt.jetzt@linus:schreibt.jetztlike if it goes down for a day, that would be much less of a problem than the ubuntu package servers being down for a day would be for ubuntu00:17:10
@linus:schreibt.jetzt@linus:schreibt.jetztergh, I should sleep, not think about this, for now00:19:46
@linus:schreibt.jetzt@linus:schreibt.jetztgood night :D00:19:48
@whentze:matrix.orgWanja Hentzeeh, for a loooot of NixOS users, the cache being down means nixos-rebuild just becomes a desk warming app00:19:53
@edef1c:matrix.orgedefyeah00:20:03
@edef1c:matrix.orgedeffrom the cache logs, i know we have a lot of users in like, low-GDP countries you've never heard of before00:20:30
@whentze:matrix.orgWanja Hentzewe can actually build our closures on our CI infra but I'm sure many can't00:20:36
@edef1c:matrix.orgedefi don't think they would appreciate the desk warming since it is coming out of their power bill00:20:46
@whentze:matrix.orgWanja Hentzeand in my hobby usage I definitely can't00:20:55

Show newer messages


Back to Room ListRoom Version: 10