If anyone has any ideas as to what could lead to a NixOS test running locally but not on a Hydra instance (the last thing that comes to my mind would be performance differences), I'd be really happy. I'll put my example on github next week, hopefully a smarter person than me can see what's going on then. The gist of it is that I'm starting the postgres service, wait for it to be up, then change the user password. This works fine.

But then doing anything after, like machine.succeed("PGPASSWORD=foobar psql -d my_db -U postgres -c 'select * from information_schemas.tables' -o blub"), I get log output that doesn't even mention the "waiting" anymore and postgres just seems to fail:

machine # [  315.423797] postgresql-pre-start[902]: WARNING: enabling "trust" authentication for local connections
machine # [  315.428066] postgresql-pre-start[902]: You can change this by editing pg_hba.conf or using the option -A, or
machine # [  315.433320] postgresql-pre-start[902]: --auth-local and --auth-host, the next time you run initdb.
machine # [  315.443602] postgresql-pre-start[902]: Success. You can now start the database server using:
machine # [  315.455771] postgresql-pre-start[902]:     /nix/store/6cn4h4icjjvg9yqx8iy55hvmqbq0jj2j-postgresql-10.21/bin/pg_ctl -D /var/lib/postgresql/10.0 -l logfile start
machine # [  315.910651] systemd[1]: postgresql.service: Failed with result 'timeout'.
machine # [  315.931785] systemd[1]: Failed to start PostgreSQL Server.
machine # [  315.949857] systemd[1]: postgresql.service: Consumed 52.412s CPU time, read 0B from disk, written 49.6M to disk, no IP traffic.
machine # [  315.996944] systemd[1]: Reached target Multi-User System.
machine # [  316.027544] systemd[1]: Startup finished in 1min 18.771s (kernel) + 3min 57.251s (userspace) = 5min 16.022s.
cleanup
kill machine (pid 8)
machine # qemu-kvm: terminating on signal 15 from pid 6 (/nix/store/vzqny68wq33dcg4hkdala51n5vqhpnwc-python3-3.9.12/bin/python3.9)
(finished: cleanup, in 0.06 seconds)
Traceback (most recent call last):
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/bin/.nixos-test-driver-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/__init__.py", line 114, in main
    driver.run_tests()
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/driver.py", line 146, in run_tests
    self.test_script()
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/driver.py", line 142, in test_script
    exec(self.tests, symbols, None)
  File "<string>", line 1, in <module>
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/machine.py", line 459, in wait_for_unit
    retry(check_active)
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/machine.py", line 126, in retry
    if fn(False):
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/machine.py", line 439, in check_active
    raise Exception('unit "{}" reached state "{}"'.format(unit, state))
Exception: unit "postgresql" reached state "failed"
kill vlan (pid 7)

In reply to @ulrikstrid:matrix.org
What is the correct way of building a nixosConfiguration?

In reply to @lewo:matrix.org
kenran_: is KVM enabled on your Hydra instance?

In reply to @kenran_:matrix.org
What do you mean exactly? A NixOS configuration with a hydra service? Your question sounds like it's about the flake output type.

In reply to @ulrikstrid:matrix.org
I manage my computers with flakes and want to have my hydra build the configurations and ideally I would deploy from that

In reply to @lewo:matrix.org
kenran_: having kvm in system-features is not enough to ensure Nix have enough permissions on /dev/kvm (if nested KVM is provided by your cloud provider)

In reply to @janne.hess:helsinki-systems.de
Adding kvm to the system features is already wrong. Nix autodetects kvm and adds the feature

In reply to @lewo:matrix.org
kenran_: having kvm in system-features is not enough to ensure Nix have enough permissions on /dev/kvm (if nested KVM is provided by your cloud provider)

In reply to @andreas.schraegle:helsinki-systems.de
vm tests should run, just very very veeeeeryyyy slow