Sadly the NixOS tests I've written fail in my Hydra instance and I don't know why. I'll wait and see if anyone answers in the NixOS channel first, then re-post here in full detail. In essence all my tests use a postgresql service which after some time leads to

machine # [  336.433468] postgresql-pre-start[896]: Success. You can now start the database server using:
machine # [  336.452698] postgresql-pre-start[896]:     /nix/store/6cn4h4icjjvg9yqx8iy55hvmqbq0jj2j-postgresql-10.21/bin/pg_ctl -D /var/lib/postgresql/10.0 -l logfile start
machine # [  337.181043] systemd[1]: postgresql.service: Failed with result 'timeout'.

In reply to @roberthensing:matrix.org
evaluation hasn't started for 13h; seems like something is wrong https://hydra.nixos.org/jobset/nixos/trunk-combined

If anyone has any ideas as to what could lead to a NixOS test running locally but not on a Hydra instance (the last thing that comes to my mind would be performance differences), I'd be really happy. I'll put my example on github next week, hopefully a smarter person than me can see what's going on then. The gist of it is that I'm starting the postgres service, wait for it to be up, then change the user password. This works fine.

But then doing anything after, like machine.succeed("PGPASSWORD=foobar psql -d my_db -U postgres -c 'select * from information_schemas.tables' -o blub"), I get log output that doesn't even mention the "waiting" anymore and postgres just seems to fail:

machine # [  315.423797] postgresql-pre-start[902]: WARNING: enabling "trust" authentication for local connections
machine # [  315.428066] postgresql-pre-start[902]: You can change this by editing pg_hba.conf or using the option -A, or
machine # [  315.433320] postgresql-pre-start[902]: --auth-local and --auth-host, the next time you run initdb.
machine # [  315.443602] postgresql-pre-start[902]: Success. You can now start the database server using:
machine # [  315.455771] postgresql-pre-start[902]:     /nix/store/6cn4h4icjjvg9yqx8iy55hvmqbq0jj2j-postgresql-10.21/bin/pg_ctl -D /var/lib/postgresql/10.0 -l logfile start
machine # [  315.910651] systemd[1]: postgresql.service: Failed with result 'timeout'.
machine # [  315.931785] systemd[1]: Failed to start PostgreSQL Server.
machine # [  315.949857] systemd[1]: postgresql.service: Consumed 52.412s CPU time, read 0B from disk, written 49.6M to disk, no IP traffic.
machine # [  315.996944] systemd[1]: Reached target Multi-User System.
machine # [  316.027544] systemd[1]: Startup finished in 1min 18.771s (kernel) + 3min 57.251s (userspace) = 5min 16.022s.
cleanup
kill machine (pid 8)
machine # qemu-kvm: terminating on signal 15 from pid 6 (/nix/store/vzqny68wq33dcg4hkdala51n5vqhpnwc-python3-3.9.12/bin/python3.9)
(finished: cleanup, in 0.06 seconds)
Traceback (most recent call last):
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/bin/.nixos-test-driver-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/__init__.py", line 114, in main
    driver.run_tests()
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/driver.py", line 146, in run_tests
    self.test_script()
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/driver.py", line 142, in test_script
    exec(self.tests, symbols, None)
  File "<string>", line 1, in <module>
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/machine.py", line 459, in wait_for_unit
    retry(check_active)
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/machine.py", line 126, in retry
    if fn(False):
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/machine.py", line 439, in check_active
    raise Exception('unit "{}" reached state "{}"'.format(unit, state))
Exception: unit "postgresql" reached state "failed"
kill vlan (pid 7)

In reply to @ulrikstrid:matrix.org
What is the correct way of building a nixosConfiguration?

In reply to @lewo:matrix.org
kenran_: is KVM enabled on your Hydra instance?

In reply to @kenran_:matrix.org
What do you mean exactly? A NixOS configuration with a hydra service? Your question sounds like it's about the flake output type.