!CcTBuBritXGywOEGWJ:matrix.org

NixOS Binary Cache Self-Hosting

171 Members
About how to host a very large-scale binary cache and more58 Servers

Load older messages


SenderMessageTime
6 Mar 2024
@delroth:delroth.netdelrothre: paying people - I'm not convinced that a self-hosted cache needs anyone to be paid to handle day to day ops (I don't view it as any different from other infra-team-managed services, especially if good DR can mitigate risks on day to day ops), but I think the main difference with existing infra work is the availability needs (which are not currently properly defined, but they exist, and I think we can all agree to say the cache hosting has higher availability requirements than anything else we self host)23:41:19
@delroth:delroth.netdelrothso maybe what we need is a model where we pay people for oncall response?23:41:32
@delroth:delroth.netdelroth disclaimer: I have no experience with such a model, I don't know that anyone does because I haven't really heard about any precedents before, but maybe this gives a direction for someone interested to try and explore 23:43:03
@edef1c:matrix.orgedefsure, though obvious necessary incentive alignment: you get paid to be responsive, not for the response23:44:06
@edef1c:matrix.orgedefie we don't pay you for the hours, you win by keeping the hours down23:44:22
@delroth:delroth.netdelrothit depends how much you think the responders are intrinsically motivated by money vs. keeping costs down - in a for-profit structure I'd definitely agree with you, in the current structure we already have tens of people spending their free time trying to reduce the foundation's spending with no expectation of personal monetary returns23:46:22
@edef1c:matrix.orgedefmmm23:46:40
@delroth:delroth.netdelrothlike, yes, I agree that in theory assuming maximally greedy humans the incentive structure I've proposed is completely broken :P23:47:03
@edef1c:matrix.orgedefmy returns are producing a public good, but i know my utility function is insane to most people23:47:53
@edef1c:matrix.orgedefobviously one can interpolate between the models a bit23:48:23
@edef1c:matrix.orgedefbroadly, relying on people with unusually high-altruism utility functions is why things are so tight23:49:37
@delroth:delroth.netdelrothsemi-related, checking my intuition: I think "reliance on people with high-altruism utility functions" is not a linear knob - I expect that once you start reaching a certain amount of people with lower altruism, you get the people with higher altruism to leave go do something else. WDYT?23:52:56
@edef1c:matrix.orgedefthere's definitely evaporative cooling effects of that nature23:53:14
@edef1c:matrix.orgedef * there are definitely evaporative cooling effects of that nature23:53:48
@delroth:delroth.netdelroth (why I think it's relevant is that if that's the case, then not relying on high-altruism people means a major shift in the current composition of the teams looking after everything) 23:53:49
@whentze:matrix.orgWanja Hentze
In reply to @edef1c:matrix.org
ie we don't pay you for the hours, you win by keeping the hours down
we do that in our on-call rotation at work and it works pretty well
23:58:35
@whentze:matrix.orgWanja Hentzewhat we do give people for on-cal hours is time off, not as an incentive but just to make sure whoever is on call is relatively well rested23:59:18
@whentze:matrix.orgWanja Hentze* what we do give people for on-call hours is time off, not as an incentive but just to make sure whoever is on call is relatively well rested23:59:26
@adam:robins.wtfadamcstephens 🐝
In reply to @delroth:delroth.net
so maybe what we need is a model where we pay people for oncall response?
I've known people who were on-call, but not salary. Not exactly the same as a situation like this, but there are models for paying people who get paged. Generally they get a minimum of a couple hours of the contract/pay rate for responding.
23:59:27
7 Mar 2024
@edef1c:matrix.orgedefalso, to be clear: if we're not paying for hours we can plausibly pay them better continuously00:00:20
@edef1c:matrix.orgedefoverall like, people in the aggregate respond to incentives in the long run, even if they short-run do otherwise or claim otherwise00:00:50
@whentze:matrix.orgWanja Hentzeand people (and especially groups of people with changing membership) can respond to incentives even without anyone consciously doing so00:01:57
@edef1c:matrix.orgedefyeah. a lot of response to incentives is by the best people leaving00:02:18
@edef1c:matrix.orgedefthe people you retain are the ones for whom the incentives work00:02:32
@edef1c:matrix.orgedefthis scene has bled people pretty hard over the ~8y i've been part of the show00:03:10
@adam:robins.wtfadamcstephens 🐝I do think we should expect to pay someone for an ongoing number of hours. While it may not be necessary to care and feed a system every day, I wouldn't underestimate the potential problems or just unforeseen maintenance of a storage cluster.00:03:13
@edef1c:matrix.orgedefpart of the thing here is, you can prevent a lot of emergencies by doing consistent work to make them unlikely and handleable00:04:27
@edef1c:matrix.orgedefspes non consilium est; hope is not a strategy00:04:49
@edef1c:matrix.orgedefconsistently building reliable systems and doing reliability work instead of responding to incidents is how we make this good00:05:19
@edef1c:matrix.orgedefand this buys us various kinds of budget to do the improving, it compounds well00:05:40

Show newer messages


Back to Room ListRoom Version: 10