!zghijEASpYQWYFzriI:nixos.org

Hydra

350 Members
96 Servers

You have reached the beginning of time (for this room).


SenderMessageTime
24 Feb 2025
@shawn8901:matrix.orgshawn8901 *

Hi, since some time i am having the following error on my hydra instance.

Everytime it does an evaluation it aborts with the following error in log:

Feb 24 17:00:38 tank nix-daemon[1579]: accepted connection from pid 6217, user hydra
Feb 24 17:00:38 tank hydra-evaluator[2106]: (config:.jobsets) Evaluating...
Feb 24 17:00:38 tank hydra-evaluator[2106]: error: stoi
Feb 24 17:00:38 tank hydra-evaluator[2106]: {UNKNOWN}: process ended prematurely at /nix/store/kvnp4qdk6bcg9j0pc8d87dgz6z5qklhl-hydra-0-unstable-2025-02-18/bin/.hydra-eval-jobset-wrapped line 404. at /nix/store/i85ni9bphygj6d31v68x24ncvhbc2vn6-hydra-perl-deps/lib/perl5/site_perl/5.40.0/Catalyst/Model/DBIC/Schema.pm line 526
Feb 24 17:00:38 tank hydra-evaluator[2078]: evaluation of jobset ‘config:.jobsets (jobset#1)’ failed with exit code 1

I am kinda out of ideas, the webserver runs fine.
I did play around a bit with the initd system (switched from scripted to systemd), if i remember correctly its kinda at the same time frame.
I noticed that there was a wrongly mapped uid for another service (which i fixed with chowning to the new id), but for hydra i did not find similar.
I also tried to reinstall hydra (nuke /var/lib/hydra and drop the database).
But all of that did not help.

I found an old issue relating to memory on that error, tho it confuses me as the machine has plenty of memory left unused and was capable to run my hydra builds before.
Has anyone an idea for me how to continue analying that issue?

edit: I found https://github.com/NixOS/hydra/issues/1437 that could be similar thing, at least the time range could fit, tho i am seing a different error text.

16:37:27
@shawn8901:matrix.orgshawn8901 Okay, its locatable to the latest hydra bump, when I revert it, it's working again fine.
Should I create an issue in nixpks or more on hydras github?
20:18:34
25 Feb 2025
@hacker1024:matrix.orghacker1024 There are only two places where nix-eval-jobs uses stoi. Have you set evaluator_workers or evaluator_max_memory_size in your Hydra configuration? 02:14:18
@shawn8901:matrix.orgshawn8901
In reply to @hacker1024:matrix.org
There are only two places where nix-eval-jobs uses stoi. Have you set evaluator_workers or evaluator_max_memory_size in your Hydra configuration?
Yeah, I am limiting that. So stoi = out of memory?
06:25:39
@hacker1024:matrix.orghacker1024 No, stoi is a function that parses an integer. What is the exact contents of that part of your config? 06:26:39
@shawn8901:matrix.orgshawn8901 evaluator_max_memory_size = ${toString (4 * 1024 * 1024 * 1024)} which was totally fine previously 06:27:35
@shawn8901:matrix.orgshawn8901 And I am setting evaluator_workers = 2 06:28:15
@hacker1024:matrix.orghacker1024Hmm yeah that does seem fine06:28:30
@hacker1024:matrix.orghacker1024Still maybe try without it for a bit and see if that helps?06:28:42

Show newer messages


Back to Room ListRoom Version: 6