!sBfrWMVsLoSyFTCkNv:nixos.org

OfBorg

161 Members
Number of builds and evals in queue: https://nix.ci/prometheus/graph?g0.expr=ofborg_queue_evaluator_waiting&g0.tab=1&g0.stacked=0&g0.show_exemplars=0&g0.range_input=2h&g1.expr=ofborg_queue_builder_waiting%7Barch!~%22.*-lowprior%22%7D&g1.tab=1&g1.stacked=0&g1.show_exemplars=0&g1.range_input=1w57 Servers

Load older messages


SenderMessageTime
4 Apr 2023
@hexa:lossy.networkhexayou just backport stuff into it and move on 13:08:11
@cole-h:matrix.orgcole-hlmao13:08:15
@raitobezarius:matrix.orgraitobezarius
In reply to @hexa:lossy.network
you just backport stuff into it and move on
greg k-h enters the channel
14:47:01
@cole-h:matrix.orgcole-hFound this thread: https://lkml.org/lkml/2023/3/16/765 So while it's not 6.2 as that thread mentions, may be the same problem14:47:34
@hexa:lossy.networkhexakernel downgrade when14:52:46
@hexa:lossy.networkhexawould also be interesting to know what its previous kernel verison was14:55:45
@hexa:lossy.networkhexa * would also be interesting to know what its previous kernel version was14:55:49
@cole-h:matrix.orgcole-h🤷 the box is unpinned, but likely 6.1.21 was its previous version14:56:08
@cole-h:matrix.orgcole-h

ok, 6.1.21 is also busted

[  110.726426] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[  110.732340] rcu:     44-...0: (0 ticks this GP) idle=9b44/1/0x4000000000000000 softirq=2863/2863 fqs=2462
[  110.741636]  (detected by 70, t=5255 jiffies, g=14289, q=8797 ncpus=80)
[  110.748238] Task dump for CPU 44:
[  110.751540] task:kworker/u160:1  state:R  running task     stack:0     pid:419   ppid:2      flags:0x0000000a
[  110.761443] Workqueue: efi_rts_wq efi_call_rts
[  110.765878] Call trace:
[  110.768312]  __switch_to+0xf0/0x170
[  110.771791]  0xffff07ff85645b80
15:38:21
@cole-h:matrix.orgcole-hnvm it's still 6.1.22 somehow15:39:29
@cole-h:matrix.orgcole-h
[  242.441034] INFO: task kworker/u160:0:9 blocked for more than 120 seconds.
[  242.447910]       Tainted: P           O       6.1.22 #1-NixOS
[  242.453735] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.461553] task:kworker/u160:0  state:D stack:0     pid:9     ppid:2      flags:0x00000008
[  242.469895] Workqueue: events_freezable_power_ sync_hw_clock
[  242.475548] Call trace:
[  242.477986]  __switch_to+0xf0/0x170
[  242.481467]  __schedule+0x30c/0x1254
[  242.485035]  schedule+0x58/0xec
[  242.488164]  schedule_timeout+0x14c/0x180
[  242.492165]  __wait_for_common+0xd4/0x250
[  242.496165]  wait_for_completion+0x28/0x34
[  242.500251]  virt_efi_set_time+0x114/0x190
[  242.504339]  efi_set_time+0x84/0xc0
[  242.507818]  rtc_set_time+0xc0/0x1c4
[  242.511385]  sync_hw_clock+0x1ac/0x230
[  242.515123]  process_one_work+0x1f4/0x460
[  242.519124]  worker_thread+0x188/0x4e0
[  242.522863]  kthread+0xe0/0xe4
[  242.525908]  ret_from_fork+0x10/0x20
15:39:43
@hexa:lossy.networkhexa
In reply to @cole-h:matrix.org
not yet (because it's not easy, if possible lol)
🤡
15:40:13
@cole-h:matrix.orgcole-hoh I missed something lol, let's try again15:55:36
@cole-h:matrix.orgcole-h
[  109.650254] rcu: rcu_sched kthread timer wakeup didn't happen for 3034 jiffies! g13841 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
[  109.661456] rcu:     Possible timer handling issue on cpu=44 timer-softirq=210
[  109.668404] rcu: rcu_sched kthread starved for 3040 jiffies! g13841 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=44
[  109.678737] rcu:     Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.
[  109.687682] rcu: RCU grace-period kthread stack dump:
[  109.692720] task:rcu_sched       state:I stack:0     pid:14    ppid:2      flags:0x00000008
[  109.701057] Call trace:
[  109.703490]  __switch_to+0xf0/0x170
[  109.706967]  __schedule+0x30c/0x1254
[  109.710530]  schedule+0x58/0xec
[  109.713659]  schedule_timeout+0xa4/0x180
[  109.717571]  rcu_gp_fqs_loop+0x138/0x4ac
[  109.721483]  rcu_gp_kthread+0x1d4/0x210
[  109.725307]  kthread+0xe0/0xe4
[  109.728350]  ret_from_fork+0x10/0x20

welp

16:22:23
@cole-h:matrix.orgcole-htrying kernel 6.1.19 (yes I skipped 6.1.20 because I didn't want to find the last nixos-unstable eval that had 6.1.20)16:31:58
@cole-h:matrix.orgcole-h

and further back we go

[  726.247262] INFO: task kworker/u160:3:519 blocked for more than 604 seconds.
[  726.254303]       Tainted: P           O       6.1.19 #1-NixOS
[  726.260124] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  726.267943] task:kworker/u160:3  state:D stack:0     pid:519   ppid:2      flags:0x00000008
[  726.276283] Workqueue: events_freezable_power_ sync_hw_clock
[  726.281933] Call trace:
[  726.284368]  __switch_to+0xf0/0x170
[  726.287848]  __schedule+0x30c/0x1254
[  726.291414]  schedule+0x58/0xec
[  726.294542]  schedule_timeout+0x14c/0x180
[  726.298542]  __wait_for_common+0xd4/0x250
[  726.302541]  wait_for_completion+0x28/0x34
[  726.306627]  virt_efi_set_time+0x114/0x190
[  726.310713]  efi_set_time+0x84/0xc0
[  726.314192]  rtc_set_time+0xc0/0x1c4
[  726.317757]  sync_hw_clock+0x1ac/0x230
[  726.321495]  process_one_work+0x1f4/0x460
[  726.325496]  worker_thread+0x188/0x4e0
[  726.329235]  kthread+0xe0/0xe4
[  726.332279]  ret_from_fork+0x10/0x20
17:14:35
@cole-h:matrix.orgcole-hmaybe I should see where the maybe-problematic commit first appeared and go to the release prior to that lol17:14:50
@hexa:lossy.networkhexainb4: this is broken since we moved stable to 6.117:15:06
@cole-h:matrix.orgcole-h😅17:15:22
@cole-h:matrix.orgcole-hwhen was that?17:15:24
@hexa:lossy.networkhexachecking17:15:35
@hexa:lossy.networkhexabut weeks17:15:38
@cole-h:matrix.orgcole-hthen it's possible17:15:48
@cole-h:matrix.orgcole-h(since I've gone back weeks now lol)17:15:59
@hexa:lossy.networkhexahence my initial question about the previous kernel 😄17:16:05
@hexa:lossy.networkhexahttps://github.com/NixOS/nixpkgs/pull/21531317:16:37
@hexa:lossy.networkhexamerged on march 3rd17:16:40
@hexa:lossy.networkhexaI assume the hydra builders are still on 5.15?17:17:42
@cole-h:matrix.orgcole-hNo idea, that is outside my domain17:19:04
@cole-h:matrix.orgcole-h Trying 10e51cdc0456f1d5c8a00f026c384f0e81126538 (the last nixos-unstable eval before 6.1 became the default) 17:19:25

There are no newer messages yet.


Back to Room ListRoom Version: 6