I noticed my hydra wasn't building yesterday,

hydra-evaluator[671730]: exception in database monitor thread: Lost connection to the database server.
hydra-evaluator[671730]: exception in main loop: Lost connection to the database server.

The db wasn't down when I looked.
Poking around the code, I see the exception messages but there's no recovery for a lost database connection. Ideally, it would reconnect. If I were to figure out a patch, should I figure out the reconnection or just abort hydra-evaluator? does the latter smell too much?

systemd would restart it, but the code carries on:

   } catch (std::exception & e) {
                  printError("exception in database monitor thread: %s", e.what());
                  sleep(30);
              }

for example.

evaluation failed with exit code 255

so this is what happens, I don't see anything useful in the stdout/stderr streams of the container (yes It's in a container)

In reply to @ksenia.portu:matrix.org
Hello, hydra website seems broken https://hydra.nixos.org/ 500 Internal Server Error

In reply to Mindavi
Nix daemon is also running in that container?

(systems:master) Evaluating...
accepted connection from pid 2869996, user hydra

It does say this

In reply to Mindavi
And normal nix build do work fine, I assume?

export PATH=${with pkgs; makeBinPath [ hydra-package nettools jq ]}:$PATH

sv -v -w 0 once hydra-init
[[ ! -e ${baseDir}/.init-hydra ]] && exit 1

export PATH=${pkgs.nettools}/bin:$PATH # Hydra runs some variant of `hostname --fqdn`, which BusyBox doesn't support
HOME=~hydra exec chpst ${optionalString cfg.adjustNiceness "-n +5"} -b hydra-evaluator -u hydra:hydra ${hydra-package}/bin/hydra-evaluator

nope, that should be what is doing the eval if I understand Hydra's architecture right