Hydra

maybe im somehow eating those logs somewhere, let me check

11:33:50

export PATH=${with pkgs; makeBinPath [ hydra-package nettools jq ]}:$PATH

sv -v -w 0 once hydra-init
[[ ! -e ${baseDir}/.init-hydra ]] && exit 1

export PATH=${pkgs.nettools}/bin:$PATH # Hydra runs some variant of `hostname --fqdn`, which BusyBox doesn't support
HOME=~hydra exec chpst ${optionalString cfg.adjustNiceness "-n +5"} -b hydra-evaluator -u hydra:hydra ${hydra-package}/bin/hydra-evaluator

nope, that should be what is doing the eval if I understand Hydra's architecture right

11:36:48

Are you not using systemd? 👀

12:16:54

That looks like linux from the 90s

im not using systemd

and it is runit so your time period guess is correct

12:19:31

Do you see any logs from the evaluator? because it should log once it starts an eval

12:20:25

(systems:master) Evaluating...
accepted connection from pid 2869996, user hydra

I only see this in the logs, first line is evaluator second is nix-daemon

12:21:35

I remember some stdout/stderr magic but I'm not sure if we still have that

12:22:49

In reply to @magic_rb:matrix.redalder.org
Hi! I'm running my own Hydra instance on a non NixOS host essentially and I've never gotten evaluation logs out of it, it just fails as shown here on this link https://hydra.redalder.org/jobset/systems/master#tabs-errors where should I start digging as to why it happens? The module I'm using is heavily based on the NixOS one though so I have no idea why it's misbehaving...

I might asking the obvious, but why is this a non nixos machine? There is probably a very good reason for this .

13:08:54

* I might be asking the obvious, but why is this a non nixos machine? There is probably a very good reason for this .

13:09:13

In reply to nixinator
* I might be asking the obvious, but why is this a non nixos machine? There is probably a very good reason for this .

not really tbh, i just decided at one point i want to run it in a container because why not.

13:10:52

I'm starting to be really surprised this works at all, I don't have git in the module mentioned anywhere. It's not in PATH, neither is Nix which is fascinating

13:41:32

nix correctly reports a failed eval from the terminal

13:45:00

This is probably how the container is handling certainly file descriptors, or something is not quite what Perl expects .

13:55:39

i've no clue, i dont see a reason why it should be misbehaving nor do i see any errors or anything like that

13:56:48

Without seeing the logs, or reproducing the bug myself there not much I can do. Also beware that some process know when they are being run from the interactive cli or spawned from unit/systemd .. so bear that in mind .

14:34:39

i sent all the logs i have, nothing indicates any failure

15:04:58

Well darn and blast . Looks like a job for dtrace then , shame they never got the licensing sorted out to use it in Linux, it’s basically the dogs bollocks.

22:52:20

Systemtap works but is a poor alternative

22:52:51

https://www.brendangregg.com/dtrace.html#OneLiners

22:55:38

This has help me nail all sorts of deep rooted bugs . :-)

22:55:54