In reply to @ivank:matrix.org
I'll try using nix-serve-ng instead of SSH substituters

or not, I guess nix-serve-ng sometimes crashes when requesting something that doesn't exist. https://gist.github.com/ivan/25faf249a5d2efa61332dd1df26eed34

(actually possibly my problem because nix-serve also has issues connecting to daemon. yeah, nix-daemon[3785473]: error: error processing connection: user '62540' is not allowed to connect to the Nix daemon so I'll go deal with that)

I noticed my hydra wasn't building yesterday,

hydra-evaluator[671730]: exception in database monitor thread: Lost connection to the database server.
hydra-evaluator[671730]: exception in main loop: Lost connection to the database server.

The db wasn't down when I looked.
Poking around the code, I see the exception messages but there's no recovery for a lost database connection. Ideally, it would reconnect. If I were to figure out a patch, should I figure out the reconnection or just abort hydra-evaluator? does the latter smell too much?

systemd would restart it, but the code carries on:

   } catch (std::exception & e) {
                  printError("exception in database monitor thread: %s", e.what());
                  sleep(30);
              }

for example.

evaluation failed with exit code 255

so this is what happens, I don't see anything useful in the stdout/stderr streams of the container (yes It's in a container)