| 21 Oct 2021 |
 K900 | https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/test-driver/test-driver.py#L957 | 19:37:14 |
| K900 | That's weird | 19:37:16 |
| K900 | Also that whole thing should probably use the JSON protocol... | 19:37:33 |
 Janne Heß | In reply to @baloo_:matrix.org
diff --git a/nixos/lib/test-driver/test-driver.py b/nixos/lib/test-driver/test-driver.py
index e659b0c04f5..1518bf3562a 100755
--- a/nixos/lib/test-driver/test-driver.py
+++ b/nixos/lib/test-driver/test-driver.py
@@ -1021,6 +1021,7 @@ class Machine:
assert self.process
assert self.shell
assert self.monitor
+ self.send_monitor_command("quit")
self.process.terminate()
self.shell.close()
self.monitor.close()
I'd give this a try
Still hangs at the end | 19:39:26 |
| K900 | Can you get a traceback? | 19:41:19 |
| Janne Heß | in python? | 19:41:28 |
| Janne Heß | py-spy should be enough? | 19:41:34 |
| K900 | Yeah | 19:41:56 |
| Janne Heß | oh lol it failed more horribly now | 19:43:00 |
| Janne Heß | clean up
kill machine (pid 9)
machine: sending monitor command: quit
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/nix/store/97w52ckcjnfiz89h3lh7zf1kysgfm2s8-python3-3.9.6/lib/python3.9/contextlib.py", line 135, in __exit__
self.gen.throw(type, value, traceback)
File "/nix/store/7v40kpl69d7ydhm2mz7wkl8q04yc8xly-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 172, in nested
yield
File "/nix/store/7v40kpl69d7ydhm2mz7wkl8q04yc8xly-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 1139, in clean_up
machine.release()
File "/nix/store/7v40kpl69d7ydhm2mz7wkl8q04yc8xly-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 1039, in release
self.send_monitor_command("quit")
File "/nix/store/7v40kpl69d7ydhm2mz7wkl8q04yc8xly-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 509, in send_monitor_command
self.monitor.send(message)
BrokenPipeError: [Errno 32] Broken pipe
| 19:43:11 |
| K900 | Huh | 19:43:21 |
| K900 | That looks like QEMU is dead | 19:43:34 |
| K900 | But not like | 19:43:40 |
| K900 | All the way dead | 19:43:42 |
| K900 | But presumably dead enough to close the monitor socket? | 19:44:02 |
| Janne Heß | it properly failed this time:
Process 43896: /nix/store/97w52ckcjnfiz89h3lh7zf1kysgfm2s8-python3-3.9.6/bin/python3 /nix/store/7v40kpl69d7ydhm2mz7wkl8q04yc8xly-nixos-test-driver/bin/.nixos-test-driver-wrapped
Python v3.9.6 (/nix/store/97w52ckcjnfiz89h3lh7zf1kysgfm2s8-python3-3.9.6/bin/python3.9)
Thread 0x7FFFF77EB340 (active)
_shutdown (threading.py:1448)
Thread 0x7FFFF6191640 (active)
process_serial_output (.nixos-test-driver-wrapped:931)
run (threading.py:910)
_bootstrap_inner (threading.py:973)
_bootstrap (threading.py:930)
| 19:45:27 |
| K900 | What if you just self.process.kill() instead of self.process.terminate()? | 19:46:27 |
| Janne Heß | hmb | 19:46:39 |
| Janne Heß | hangs again, same stack | 19:48:43 |
| K900 | Does the process get killed? | 19:48:55 |
| Janne Heß | I already killed the runner, wait a minute | 19:49:04 |
| Janne Heß | (or two…) | 19:49:08 |
| K900 | If QEMU is ignoring SIGKILL, we're extremely fucked tbh | 19:49:43 |
| Janne Heß | qemu is still there | 19:52:12 |
| K900 | Uhh | 19:52:36 |
| K900 | That is not a good sign | 19:52:40 |
| K900 | Stack trace? | 19:52:47 |
| Janne Heß | qemu or python? | 19:52:57 |
| K900 | QEMU | 19:53:10 |
| Janne Heß | /proc/44797/task/44797/stack
[<0>] do_sys_poll+0x3ab/0x5b0
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x33/0x40
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
/proc/44797/task/44805/stack
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe9/0x240
[<0>] do_futex+0x174/0xbf0
[<0>] __x64_sys_futex+0x146/0x1c0
[<0>] do_syscall_64+0x33/0x40
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
/proc/44797/task/44810/stack
[<0>] kvm_vcpu_block+0x58/0x2f0 [kvm]
[<0>] kvm_arch_vcpu_ioctl_run+0x6c4/0x1720 [kvm]
[<0>] kvm_vcpu_ioctl+0x211/0x5a0 [kvm]
[<0>] __x64_sys_ioctl+0x83/0xb0
[<0>] do_syscall_64+0x33/0x40
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
/proc/44797/stack
[<0>] do_sys_poll+0x3ab/0x5b0
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x33/0x40
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
| 19:53:31 |
