In reply to @jackdk:matrix.org
But because the flake completely specifies its inputs, ISTM that you don't need the separate spec.json to declare which inputs you want in NIX_PATH or whatever

Sadly the NixOS tests I've written fail in my Hydra instance and I don't know why. I'll wait and see if anyone answers in the NixOS channel first, then re-post here in full detail. In essence all my tests use a postgresql service which after some time leads to

machine # [  336.433468] postgresql-pre-start[896]: Success. You can now start the database server using:
machine # [  336.452698] postgresql-pre-start[896]:     /nix/store/6cn4h4icjjvg9yqx8iy55hvmqbq0jj2j-postgresql-10.21/bin/pg_ctl -D /var/lib/postgresql/10.0 -l logfile start
machine # [  337.181043] systemd[1]: postgresql.service: Failed with result 'timeout'.

In reply to @roberthensing:matrix.org
evaluation hasn't started for 13h; seems like something is wrong https://hydra.nixos.org/jobset/nixos/trunk-combined

If anyone has any ideas as to what could lead to a NixOS test running locally but not on a Hydra instance (the last thing that comes to my mind would be performance differences), I'd be really happy. I'll put my example on github next week, hopefully a smarter person than me can see what's going on then. The gist of it is that I'm starting the postgres service, wait for it to be up, then change the user password. This works fine.

But then doing anything after, like machine.succeed("PGPASSWORD=foobar psql -d my_db -U postgres -c 'select * from information_schemas.tables' -o blub"), I get log output that doesn't even mention the "waiting" anymore and postgres just seems to fail:

machine # [  315.423797] postgresql-pre-start[902]: WARNING: enabling "trust" authentication for local connections
machine # [  315.428066] postgresql-pre-start[902]: You can change this by editing pg_hba.conf or using the option -A, or
machine # [  315.433320] postgresql-pre-start[902]: --auth-local and --auth-host, the next time you run initdb.
machine # [  315.443602] postgresql-pre-start[902]: Success. You can now start the database server using:
machine # [  315.455771] postgresql-pre-start[902]:     /nix/store/6cn4h4icjjvg9yqx8iy55hvmqbq0jj2j-postgresql-10.21/bin/pg_ctl -D /var/lib/postgresql/10.0 -l logfile start
machine # [  315.910651] systemd[1]: postgresql.service: Failed with result 'timeout'.
machine # [  315.931785] systemd[1]: Failed to start PostgreSQL Server.
machine # [  315.949857] systemd[1]: postgresql.service: Consumed 52.412s CPU time, read 0B from disk, written 49.6M to disk, no IP traffic.
machine # [  315.996944] systemd[1]: Reached target Multi-User System.
machine # [  316.027544] systemd[1]: Startup finished in 1min 18.771s (kernel) + 3min 57.251s (userspace) = 5min 16.022s.
cleanup
kill machine (pid 8)
machine # qemu-kvm: terminating on signal 15 from pid 6 (/nix/store/vzqny68wq33dcg4hkdala51n5vqhpnwc-python3-3.9.12/bin/python3.9)
(finished: cleanup, in 0.06 seconds)
Traceback (most recent call last):
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/bin/.nixos-test-driver-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/__init__.py", line 114, in main
    driver.run_tests()
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/driver.py", line 146, in run_tests
    self.test_script()
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/driver.py", line 142, in test_script
    exec(self.tests, symbols, None)
  File "<string>", line 1, in <module>
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/machine.py", line 459, in wait_for_unit
    retry(check_active)
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/machine.py", line 126, in retry
    if fn(False):
  File "/nix/store/hxkw9fj8l3ipraz0q8gkfalv35a6vh4v-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/machine.py", line 439, in check_active
    raise Exception('unit "{}" reached state "{}"'.format(unit, state))
Exception: unit "postgresql" reached state "failed"
kill vlan (pid 7)