Hydra

Hi! I'm running my own Hydra instance on a non NixOS host essentially and I've never gotten evaluation logs out of it, it just fails as shown here on this link https://hydra.redalder.org/jobset/systems/master#tabs-errors where should I start digging as to why it happens? The module I'm using is heavily based on the NixOS one though so I have no idea why it's misbehaving...

10:14:26

Redacted or Malformed Event

10:18:20

evaluation failed with exit code 255

so this is what happens, I don't see anything useful in the stdout/stderr streams of the container (yes It's in a container)

10:20:57

Nix daemon is also running in that container?

10:21:31

ajs124

In reply to @ksenia.portu:matrix.org
Hello, hydra website seems broken https://hydra.nixos.org/ 500 Internal Server Error

-> #infra:nixos.org

10:21:39

In reply to Mindavi
Nix daemon is also running in that container?

yep

10:21:54

And normal nix build do work fine, I assume?

10:22:24

(systems:master) Evaluating...
accepted connection from pid 2869996, user hydra

It does say this

10:22:54

In reply to Mindavi
And normal nix build do work fine, I assume?

yeah once it evals fine it builds fine and build output also works

10:23:10

Not sure what is up then, I would expect some more logs...

11:33:17

There is a PR open to add logging to all die calls IIRC

11:33:33

Might be worth trying that, if you can find it

11:33:46

maybe im somehow eating those logs somewhere, let me check

11:33:50

export PATH=${with pkgs; makeBinPath [ hydra-package nettools jq ]}:$PATH

sv -v -w 0 once hydra-init
[[ ! -e ${baseDir}/.init-hydra ]] && exit 1

export PATH=${pkgs.nettools}/bin:$PATH # Hydra runs some variant of `hostname --fqdn`, which BusyBox doesn't support
HOME=~hydra exec chpst ${optionalString cfg.adjustNiceness "-n +5"} -b hydra-evaluator -u hydra:hydra ${hydra-package}/bin/hydra-evaluator

nope, that should be what is doing the eval if I understand Hydra's architecture right

11:36:48

Are you not using systemd? 👀

12:16:54

That looks like linux from the 90s

im not using systemd

and it is runit so your time period guess is correct

12:19:31

Do you see any logs from the evaluator? because it should log once it starts an eval

12:20:25

(systems:master) Evaluating...
accepted connection from pid 2869996, user hydra

I only see this in the logs, first line is evaluator second is nix-daemon

12:21:35

I remember some stdout/stderr magic but I'm not sure if we still have that

12:22:49

In reply to @magic_rb:matrix.redalder.org
Hi! I'm running my own Hydra instance on a non NixOS host essentially and I've never gotten evaluation logs out of it, it just fails as shown here on this link https://hydra.redalder.org/jobset/systems/master#tabs-errors where should I start digging as to why it happens? The module I'm using is heavily based on the NixOS one though so I have no idea why it's misbehaving...

I might asking the obvious, but why is this a non nixos machine? There is probably a very good reason for this .

13:08:54

* I might be asking the obvious, but why is this a non nixos machine? There is probably a very good reason for this .

13:09:13

In reply to nixinator
* I might be asking the obvious, but why is this a non nixos machine? There is probably a very good reason for this .

not really tbh, i just decided at one point i want to run it in a container because why not.

13:10:52

I'm starting to be really surprised this works at all, I don't have git in the module mentioned anywhere. It's not in PATH, neither is Nix which is fascinating

13:41:32

nix correctly reports a failed eval from the terminal

13:45:00

This is probably how the container is handling certainly file descriptors, or something is not quite what Perl expects .

13:55:39

i've no clue, i dont see a reason why it should be misbehaving nor do i see any errors or anything like that

13:56:48