!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

396 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.120 Servers

Load older messages


SenderMessageTime
12 Aug 2021
@vcunat:matrix.orgVladimír ČunátStill, wendy's age has many disadvantages. Poor performance (of each single thread) and power efficiency come to mind.11:40:52
@hexa:lossy.networkhexasurprising people with tensorflow/pytorch builds on stable upgrades sadly isn't a great experience11:44:32
@hexa:lossy.networkhexaI do acknowledge that we would loose a bunch of compute, that somewhat diversifies our builder situation (eqx metal + X)11:46:40
@vcunat:matrix.orgVladimír Čunát Diversity: I've been considering to host some better physical machine than the current t4a, perhaps as a replacement of t4a. 11:48:37
@vcunat:matrix.orgVladimír ČunátIf there's interest, that is. I have no idea of any existing plans around this.11:50:08
@vcunat:matrix.orgVladimír Čunát * Diversity: I've been considering to host some better physical machine than the current t4b, perhaps as a replacement of t4b. 12:58:56
@domenkozar:matrix.orgDomen Kožar hexa: we wouldn't lose that much with wendy gone 13:47:52
@domenkozar:matrix.orgDomen Kožarit probably just needs to be done13:48:06
@andi:kack.itandi-It would still be nice to think about diversifying the builders. If packet pulls the plug (for whatever reason) we have a problem.13:49:25
@sternenseemann:systemli.orgsterniindeed13:50:52
@vcunat:matrix.orgVladimír Čunát I think ike is also from the old set of machines. 14:01:31
@vcunat:matrix.orgVladimír ČunátAs for pulling the plug, I think the main problem is in the parts that are hard to decentralize (e.g. queue runner)14:03:14
@lukegb:zxcvbnm.ninjalukegb (he/him)I don't think it's a case of "we need always running decentralized infrastructure"15:15:12
@lukegb:zxcvbnm.ninjalukegb (he/him)It's more a case of "what is the plan if packet goes away"15:15:24
@sternenseemann:systemli.orgsternipacket doesn't even have to walk away15:16:52
@sternenseemann:systemli.orgsterniwhat if they have a serious network or power outtage or even … uh a fire15:17:03
@vcunat:matrix.orgVladimír ČunátNetwork or power outage shouldn't last long enough to really hurt us. Cache etc. is separate.15:18:19
@lukegb:zxcvbnm.ninjalukegb (he/him)Hydra isn't really a service we need to run at 5 9s or anything15:18:45
@amine.chikhaoui:matrix.orgAmine ChikhaouiI think there is enough budget to cover unforeseen problems ? (https://opencollective.com/nixos#category-BUDGET) until a plan is figured out for when something unexpected happens15:21:47
@sternenseemann:systemli.orgsterniI don't think having the money is enough to be honest16:06:34
@sternenseemann:systemli.orgsternithe freenode -> matrix switch has if anything shown that if you have to scramble to find a solution in the moment the outcome will not be 100% satisfying and there is no room to get consensus for the drawbacks (which is probably the most problematic about this)16:07:41
14 Aug 2021
@lukegb:zxcvbnm.ninjalukegb (he/him)bringing up https://github.com/NixOS/nixpkgs/issues/115425 again; can we disable wendy on hydra please? (set maxjobs back to 0)14:36:10
@sternenseemann:systemli.orgsterniwhy don't we have a feature for sse 4.2 or such? isn't that precisely what that field is for?14:38:02
@lukegb:zxcvbnm.ninjalukegb (he/him)wendy just doesn't do very well on big-parallel jobs anyway (e.g. Chrome builds)14:39:20
@lukegb:zxcvbnm.ninjalukegb (he/him)if we want to eke all the life we can from wendy then we should remove the big-parallel required feature and add features for processor extensions to all the builders that do have them14:40:36
@lukegb:zxcvbnm.ninjalukegb (he/him)but at the moment: the work wendy is doing could be done faster and better by one of the other workers14:41:14
@vcunat:matrix.orgVladimír Čunát My understanding is that wendy got switched to big-parallel jobs exactly in order to avoid pytorch. 15:35:53
@lukegb:zxcvbnm.ninjalukegb (he/him)Yeah, but then I made pytorch big-parallel because the build takes a long time and is parallelisable xD15:43:18
@vcunat:matrix.orgVladimír ČunátOh, I didn't notice the reply and found out independently.15:51:25
@vcunat:matrix.orgVladimír ČunátFrom what I've seen, wendy really is more trouble than worth.15:53:03

Show newer messages


Back to Room ListRoom Version: 6