!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

274 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.91 Servers

Load older messages


SenderMessageTime
12 Aug 2021
@hexa:lossy.networkhexahttps://hydra.nixos.org/build/14956554611:25:30
@hexa:lossy.networkhexa * I have no particular love for wendy and the ongoing SIGILL problems on packages that are costly to build yourself (pytorch, tensorflow, numpy) are less than ideal11:25:55
@vcunat:matrix.orgVladimír ČunátWell, there are two sides. Producing packages that are only usable on new CPUs isn't ideal by itself... but that's perhaps something not well supported upstream. Old-style packages always compiled multiple variants where it mattered and chose during startup.11:38:36
@vcunat:matrix.orgVladimír Čunát * Well, there are two sides. Producing packages that are only usable on new CPUs isn't ideal by itself... but that's maybe something not well supported upstream (for pytorch and some others). Old-style packages always compiled multiple variants where it mattered and chose during startup.11:38:59
@vcunat:matrix.orgVladimír ČunátStill, wendy's age has many disadvantages. Poor performance (of each single thread) and power efficiency come to mind.11:40:52
@hexa:lossy.networkhexasurprising people with tensorflow/pytorch builds on stable upgrades sadly isn't a great experience11:44:32
@hexa:lossy.networkhexaI do acknowledge that we would loose a bunch of compute, that somewhat diversifies our builder situation (eqx metal + X)11:46:40
@vcunat:matrix.orgVladimír Čunát Diversity: I've been considering to host some better physical machine than the current t4a, perhaps as a replacement of t4a. 11:48:37
@vcunat:matrix.orgVladimír ČunátIf there's interest, that is. I have no idea of any existing plans around this.11:50:08
@vcunat:matrix.orgVladimír Čunát * Diversity: I've been considering to host some better physical machine than the current t4b, perhaps as a replacement of t4b. 12:58:56
@domenkozar:matrix.orgDomen Kožar hexa: we wouldn't lose that much with wendy gone 13:47:52
@domenkozar:matrix.orgDomen Kožarit probably just needs to be done13:48:06
@andi:kack.itandi-It would still be nice to think about diversifying the builders. If packet pulls the plug (for whatever reason) we have a problem.13:49:25
@sternenseemann:systemli.orgsterni (he/him)indeed13:50:52
@vcunat:matrix.orgVladimír Čunát I think ike is also from the old set of machines. 14:01:31
@vcunat:matrix.orgVladimír ČunátAs for pulling the plug, I think the main problem is in the parts that are hard to decentralize (e.g. queue runner)14:03:14
@lukegb:zxcvbnm.ninjalukegb (he/him)I don't think it's a case of "we need always running decentralized infrastructure"15:15:12
@lukegb:zxcvbnm.ninjalukegb (he/him)It's more a case of "what is the plan if packet goes away"15:15:24
@sternenseemann:systemli.orgsterni (he/him)packet doesn't even have to walk away15:16:52
@sternenseemann:systemli.orgsterni (he/him)what if they have a serious network or power outtage or even … uh a fire15:17:03
@vcunat:matrix.orgVladimír ČunátNetwork or power outage shouldn't last long enough to really hurt us. Cache etc. is separate.15:18:19
@lukegb:zxcvbnm.ninjalukegb (he/him)Hydra isn't really a service we need to run at 5 9s or anything15:18:45
@amine.chikhaoui:matrix.orgAmine ChikhaouiI think there is enough budget to cover unforeseen problems ? (https://opencollective.com/nixos#category-BUDGET) until a plan is figured out for when something unexpected happens15:21:47
@sternenseemann:systemli.orgsterni (he/him)I don't think having the money is enough to be honest16:06:34
@sternenseemann:systemli.orgsterni (he/him)the freenode -> matrix switch has if anything shown that if you have to scramble to find a solution in the moment the outcome will not be 100% satisfying and there is no room to get consensus for the drawbacks (which is probably the most problematic about this)16:07:41
14 Aug 2021
@lukegb:zxcvbnm.ninjalukegb (he/him)bringing up https://github.com/NixOS/nixpkgs/issues/115425 again; can we disable wendy on hydra please? (set maxjobs back to 0)14:36:10
@sternenseemann:systemli.orgsterni (he/him)why don't we have a feature for sse 4.2 or such? isn't that precisely what that field is for?14:38:02
@lukegb:zxcvbnm.ninjalukegb (he/him)wendy just doesn't do very well on big-parallel jobs anyway (e.g. Chrome builds)14:39:20
@lukegb:zxcvbnm.ninjalukegb (he/him)if we want to eke all the life we can from wendy then we should remove the big-parallel required feature and add features for processor extensions to all the builders that do have them14:40:36
@lukegb:zxcvbnm.ninjalukegb (he/him)but at the moment: the work wendy is doing could be done faster and better by one of the other workers14:41:14

There are no newer messages yet.


Back to Room ListRoom Version: 6