!CcTBuBritXGywOEGWJ:matrix.org

NixOS Binary Cache Self-Hosting

172 Members
About how to host a very large-scale binary cache and more59 Servers

Load older messages


SenderMessageTime
6 Mar 2024
@whentze:matrix.orgWanja Hentze* what we do give people for on-call hours is time off, not as an incentive but just to make sure whoever is on call is relatively well rested23:59:26
@adam:robins.wtfadamcstephens 🐝
In reply to @delroth:delroth.net
so maybe what we need is a model where we pay people for oncall response?
I've known people who were on-call, but not salary. Not exactly the same as a situation like this, but there are models for paying people who get paged. Generally they get a minimum of a couple hours of the contract/pay rate for responding.
23:59:27
7 Mar 2024
@edef1c:matrix.orgedefalso, to be clear: if we're not paying for hours we can plausibly pay them better continuously00:00:20
@edef1c:matrix.orgedefoverall like, people in the aggregate respond to incentives in the long run, even if they short-run do otherwise or claim otherwise00:00:50
@whentze:matrix.orgWanja Hentzeand people (and especially groups of people with changing membership) can respond to incentives even without anyone consciously doing so00:01:57
@edef1c:matrix.orgedefyeah. a lot of response to incentives is by the best people leaving00:02:18
@edef1c:matrix.orgedefthe people you retain are the ones for whom the incentives work00:02:32
@edef1c:matrix.orgedefthis scene has bled people pretty hard over the ~8y i've been part of the show00:03:10
@adam:robins.wtfadamcstephens 🐝I do think we should expect to pay someone for an ongoing number of hours. While it may not be necessary to care and feed a system every day, I wouldn't underestimate the potential problems or just unforeseen maintenance of a storage cluster.00:03:13
@edef1c:matrix.orgedefpart of the thing here is, you can prevent a lot of emergencies by doing consistent work to make them unlikely and handleable00:04:27
@edef1c:matrix.orgedefspes non consilium est; hope is not a strategy00:04:49
@edef1c:matrix.orgedefconsistently building reliable systems and doing reliability work instead of responding to incidents is how we make this good00:05:19
@edef1c:matrix.orgedefand this buys us various kinds of budget to do the improving, it compounds well00:05:40
@edef1c:matrix.orgedeflike, what i want from the incentive alignment is avoidance of "no, we won't pay you to spend time hacking on making this system more stable, but we will pay you to respond to the emergencies this creates"00:06:25
@adam:robins.wtfadamcstephens 🐝makes sense to me00:07:20
@whentze:matrix.orgWanja Hentze
In reply to @edef1c:matrix.org
part of the thing here is, you can prevent a lot of emergencies by doing consistent work to make them unlikely and handleable
yes. I'm currently on-call. The last time I was paged was months ago, and it was a false positive. the time before that was years ago.
00:07:30
@adam:robins.wtfadamcstephens 🐝that's the dream rotation00:07:45
@raitobezarius:matrix.orgraitobezarius
In reply to @whentze:matrix.org
yes. I'm currently on-call. The last time I was paged was months ago, and it was a false positive. the time before that was years ago.
do we need to make things spicier in nixos
00:07:51
@raitobezarius:matrix.orgraitobezariussorry00:07:53
@edef1c:matrix.orgedef
In reply to @whentze:matrix.org
yes. I'm currently on-call. The last time I was paged was months ago, and it was a false positive. the time before that was years ago.
and thus it would be a reasonable bet for me to take that you work in an environment where those incentives are well-aligned
00:08:01
@adam:robins.wtfadamcstephens 🐝i've definitely had on-call rotations that were much more painful than a couple times a year :)00:08:09
@whentze:matrix.orgWanja Hentzeif you do a good job on the infra and have reasonable requirements (this one is important), on-call can be relatively okay00:08:19
@whentze:matrix.orgWanja Hentzethat being said I would currently not join a 2nd rotation because I simply need time off for real once in a while00:11:49
@whentze:matrix.orgWanja HentzeI can help with infra work though if you want me to beat a NixOS over the head or replace a hard drive or whatever00:13:21
@linus:schreibt.jetzt@linus:schreibt.jetztYeah I'm not sure "on-call" makes that much sense here anyway? I'd consider it to be a best-effort thing, and I expect that if stuff catches fire people will scramble to fix it, but nobody needs to be paged out of whatever they're doing for it00:15:25
@whentze:matrix.orgWanja Hentzeidk, crates.io has a rotation and I consider that less essential00:16:07
@linus:schreibt.jetzt@linus:schreibt.jetzt we also have the advantage of the cache not being super critical for nix/nixos to be usable 00:16:16
@linus:schreibt.jetzt@linus:schreibt.jetztlike if it goes down for a day, that would be much less of a problem than the ubuntu package servers being down for a day would be for ubuntu00:17:10
@linus:schreibt.jetzt@linus:schreibt.jetztergh, I should sleep, not think about this, for now00:19:46
@linus:schreibt.jetzt@linus:schreibt.jetztgood night :D00:19:48

Show newer messages


Back to Room ListRoom Version: 10