| 4 Apr 2023 |
 cole-h | This shows the number of waiting builds and evals. | 01:59:18 |
| cole-h | In reply to @hexa:lossy.network
trace: lib.zip is deprecated, use lib.zipAttrsWith instead
trace: warning: lib.readPathsFromFile is deprecated, use a list instead
trace: warning: replaceChars is a deprecated alias of replaceStrings, replace usages of it with replaceStrings.
trace: `lib.nixpkgsVersion` is deprecated, use `lib.version` instead!
trace: lib.crossLists is deprecated, use lib.cartesianProductOfSets instead
trace: warning: literalExample is deprecated, use literalExpression instead, or use literalDocBook for a non-Nix description.
error: reading from file: Connection reset by peer
… while evaluating the attribute 'installPhase' of the derivation 'fgl-5.7.0.3'
As for this, I'll investigate in the morning :/// | 01:59:41 |
 hexa | thanks | 02:01:13 |
|  @grahamc:nixos.orgchanged room power levels. | 02:04:05 |
| cole-h set the room topic to "Number of builds and evals in queue: https://nix.ci/prometheus/graph?g0.expr=ofborg_queue_evaluator_waiting&g0.tab=1&g0.stacked=0&g0.show_exemplars=0&g0.range_input=2h&g1.expr=ofborg_queue_builder_waiting%7Barch!~%22.*-lowprior%22%7D&g1.tab=1&g1.stacked=0&g1.show_exemplars=0&g1.range_input=1w". | 02:04:44 |
| hexa | cool! | 02:10:14 |
| hexa | the aarch64 community builder is down, so no aarch64-linux build results right now | 12:39:38 |
| cole-h | I've rebooted it | 12:52:14 |
| hexa | ssh is very slow 🤔 | 13:03:20 |
| hexa | stuck at kexinit | 13:05:12 |
| hexa | if I didn't know any better I would assume MTU 😛 | 13:05:30 |
| cole-h | [ 550.001721] mlx5_core 0001:01:00.1: wait_func:1137:(pid 18057): MODIFY_CQ(0x403) canceled on out of queue timeout.
[ 550.001723] mlx5_core 0001:01:00.0: wait_func:1137:(pid 18053): MODIFY_CQ(0x403) canceled on out of queue timeout.
[ 551.221694] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[ 551.227603] rcu: 60-...0: (4 GPs behind) idle=d9e4/1/0x4000000000000000 softirq=1016/1016 fqs=54368
[ 551.236726] (detected by 34, t=115624 jiffies, g=13597, q=153057 ncpus=80)
[ 551.243675] Task dump for CPU 60:
[ 551.246977] task:kworker/u160:5 state:R running task stack:0 pid:815 ppid:2 flags:0x0000000a
[ 551.256879] Workqueue: efi_rts_wq efi_call_rts
[ 551.261313] Call trace:
[ 551.263747] __switch_to+0xf0/0x170
[ 551.267226] 0xffff081f5b486ac0
[ 556.145647] mlx5_core 0001:01:00.0: wait_func:1137:(pid 18065): ACCESS_REG(0x805) canceled on out of queue timeout.
[ 558.193622] mlx5_core 0001:01:00.0: wait_func:1137:(pid 18068): ACCESS_REG(0x805) canceled on out of queue timeout.
| 13:05:34 |
| cole-h | lol | 13:05:36 |
| hexa | low entropy? | 13:05:38 |
| hexa | that call trace is magnificent | 13:06:10 |
| hexa | __switch_to! | 13:06:14 |
| cole-h | [ 605.297123] INFO: task kworker/u160:3:519 blocked for more than 483 seconds.
[ 605.324779] Tainted: P O 6.1.22 #1-NixOS
[ 605.330601] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 605.338417] task:kworker/u160:3 state:D stack:0 pid:519 ppid:2 flags:0x00000008
[ 605.346758] Workqueue: events_freezable_power_ sync_hw_clock
[ 605.352409] Call trace:
[ 605.354844] __switch_to+0xf0/0x170
[ 605.358325] __schedule+0x30c/0x1254
[ 605.361889] schedule+0x58/0xec
[ 605.365017] schedule_timeout+0x14c/0x180
[ 605.369017] __wait_for_common+0xd4/0x250
[ 605.373017] wait_for_completion+0x28/0x34
[ 605.377102] virt_efi_set_time+0x114/0x190
[ 605.381188] efi_set_time+0x84/0xc0
[ 605.384664] rtc_set_time+0xc0/0x1c4
[ 605.388229] sync_hw_clock+0x1ac/0x230
[ 605.391966] process_one_work+0x1f4/0x460
[ 605.395966] worker_thread+0x188/0x4e0
[ 605.399704] kthread+0xe0/0xe4
[ 605.402747] ret_from_fork+0x10/0x20
[ 605.406326] INFO: task kworker/7:1H:808 blocked for more than 362 seconds.
[ 605.413189] Tainted: P O 6.1.22 #1-NixOS
[ 605.419009] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 605.426826] task:kworker/7:1H state:D stack:0 pid:808 ppid:2 flags:0x00000008
[ 605.435165] Workqueue: kblockd blk_mq_timeout_work
[ 605.439946] Call trace:
[ 605.442381] __switch_to+0xf0/0x170
[ 605.445858] __schedule+0x30c/0x1254
[ 605.449423] schedule+0x58/0xec
[ 605.452551] schedule_timeout+0x14c/0x180
[ 605.456550] __wait_for_common+0xd4/0x250
[ 605.460548] wait_for_completion+0x28/0x34
[ 605.464633] __wait_rcu_gp+0x194/0x1c4
[ 605.468371] synchronize_rcu+0x68/0xa0
[ 605.472110] blk_mq_timeout_work+0x198/0x1dc
[ 605.476369] process_one_work+0x1f4/0x460
[ 605.480368] worker_thread+0x188/0x4e0
[ 605.484106] kthread+0xe0/0xe4
[ 605.487149] ret_from_fork+0x10/0x20
| 13:06:14 |
| cole-h | I'm gonna bonk it again | 13:06:39 |
| hexa | want to try a previous regeneration? | 13:06:59 |
| hexa | if you even have that 😄 | 13:07:08 |
| cole-h | not yet (because it's not easy, if possible lol) | 13:07:16 |
| cole-h | telling Equinix to reboot the box is much easier hehe | 13:07:30 |
| hexa | could very well be a kernel regression | 13:07:31 |
| cole-h | lovely | 13:07:39 |
| hexa | because who tests lts kernels, right? | 13:08:03 |
