!zghijEASpYQWYFzriI:nixos.org

Hydra

395 Members
111 Servers

Load older messages


SenderMessageTime
14 Jan 2022
@grahamc:nixos.org@grahamc:nixos.orgdespite knowing there is a bug in this code, and that I should refactor and write tests anyway, my heart is not in it: https://github.com/DeterminateSystems/hydra/pull/6/files14:31:58
@grahamc:nixos.org@grahamc:nixos.orgI found a URL on hydra.nixos.org that causes 750Mb/s of data transfer from the database to the webserver, sustained, for several minutes: https://monitoring.nixos.org/grafana/d/rYdddlPWk/node-exporter-full?orgId=1&refresh=30s&from=now-15m&to=now&var-DS_PROMETHEUS=default&var-job=node&var-node=haumea%3A9100&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B17:40:23
@grahamc:nixos.org@grahamc:nixos.orgpretty "cool"17:40:29
@grahamc:nixos.org@grahamc:nixos.organd memory is getting squeezed on the receiver: https://monitoring.nixos.org/grafana/d/rYdddlPWk/node-exporter-full?orgId=1&refresh=30s&from=now-5m&to=now&var-DS_PROMETHEUS=default&var-job=node&var-node=ceres:9100&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B17:41:16
@casey:hubns.netcransomquery that that doesn't paginate or just throws back most of the database, eh?17:42:06
@grahamc:nixos.org@grahamc:nixos.orgyes-and!17:43:27
@casey:hubns.netcransomhydra master of improv.17:43:56
@andreas.schraegle:helsinki-systems.deajs124I deleted a full nixpkgs eval jobset from my hydra yesterday and the query took >1h. Maybe my postgres isn't configured properly 🤷17:44:22
@grahamc:nixos.org@grahamc:nixos.orghydra.nixos.org disabled deleting jobsets because of that17:45:28
@grahamc:nixos.org@grahamc:nixos.org I think cleaning up some FK's and indexes will make that much faster 17:45:39
@grahamc:nixos.org@grahamc:nixos.orgon that note, https://github.com/NixOS/hydra/pull/1109 brings us closer to doing that17:56:11
@grahamc:nixos.org@grahamc:nixos.orgTIL you can have github trigger evaluations on push19:45:25
@grahamc:nixos.org@grahamc:nixos.organd it works on flakes: https://github.com/NixOS/hydra/pull/1110/commits/ee529c6c33b37ef76e33db685fe7fcc32312c18719:46:40
15 Jan 2022
@grahamc:nixos.org@grahamc:nixos.org
In reply to @andreas.schraegle:helsinki-systems.de
I tried to delete a project right now and this happened:
https://github.com/NixOS/hydra/pull/1114
01:39:21
@grahamc:nixos.org@grahamc:nixos.organd this PR is getting smaller: https://github.com/NixOS/hydra/pull/109301:49:00
@grahamc:nixos.org@grahamc:nixos.orgchipping away at it, making PRs for each little bit01:49:07
17 Jan 2022
@grahamc:nixos.org@grahamc:nixos.orgmerged & deployed and it seems to be going fine :) whose next? 😉16:00:20
18 Jan 2022
@grahamc:nixos.org@grahamc:nixos.org

anyone seen errors like this one before?

20:21:19
@grahamc:nixos.org@grahamc:nixos.org *

anyone seen errors like this one before?

possibly transient failure building ‘/nix/store/xxx-stage-1-init.sh.drv’ on ‘localhost’: dependency '/nix/store/xxx-extra-utils' of '/nix/store/xxx-stage-1-init.sh.drv' does not exist, and substitution is disabled

20:21:39
@grahamc:nixos.org@grahamc:nixos.orgshouldn't hydra 's queue runner be responsible for sending all the inputs?20:22:29
@grahamc:nixos.org@grahamc:nixos.orgoh no this is when I go ham on the queue runner isn't it20:33:45
@grahamc:nixos.org@grahamc:nixos.orgI hate this21:04:43
19 Jan 2022
@grahamc:nixos.org@grahamc:nixos.orgthis is interesting, the queue runner when creating the build step considers the derivation for the missing output but then doesn't do anything with it16:18:26
@grahamc:nixos.org@grahamc:nixos.org

what I've found is it is checking to see if the destStore (an S3BinaryCacheStoreImpl) has it as a valid path:

State::createStep (this=0x7fffffff7280, destStore=..., conn=..., build=std::shared_ptr<Build> (use count 5, weak count 1) = {...}, drvPath=..., referringBuild=std::shared_ptr<Build> (empty) = {...}, 
    referringStep=std::shared_ptr<Step> (use count 2, weak count 2) = {...}, finishedDrvs=std::set with 1 element = {...}, newSteps=std::set with 0 elements, newRunnable=std::set with 0 elements)
    at queue-monitor.cc:470
470	        if (!destStore->isValidPath(*i.second.path(*localStore, step->drv->name, i.first))) {

isValidPath checks the disk cache:

   428      if (diskCache) {
   429          auto res = diskCache->lookupNarInfo(getUri(), hashPart);
   430          if (res.first != NarInfoDiskCache::oUnknown) {

it doesn't know (oUnknown) and falls through to:

   439      bool valid = isValidPathUncached(storePath);
   440  
   441      if (diskCache && !valid)
   442          // FIXME: handle valid = true case.
   443          diskCache->upsertNarInfo(getUri(), hashPart, 0);
   444  
   445      return valid;

and it does exist which ends up returning valid (true) here, with that FIXME hint.

from there we're back in the queue monitor:

   470          if (!destStore->isValidPath(*i.second.path(*localStore, step->drv->name, i.first))) {
   471              valid = false;
   472              missing.insert_or_assign(i.first, i.second);
   473          }

except the path is valid on the remote end despite not being on the local machine, so it never gets put in to "missing", so it never enters this:

   475      /* Try to copy the missing paths from the local store or from
   476         substitutes. */
   477      if (!missing.empty()) {

at which point Hydra thinks it is ready to build a derivation where a dependent derivation's outputs are not available

21:15:42
@grahamc:nixos.org@grahamc:nixos.org but I don't understand why it does work, substituting from the cache, in many ways. 21:19:59
@grahamc:nixos.org@grahamc:nixos.org * but I don't understand why it does work, substituting from the cache, in many cases. 21:20:01
@grahamc:nixos.org@grahamc:nixos.orgif this rings any bells at all I'd be indebted 21:26:50
20 Jan 2022
@andi:kack.itandi- left the room.00:14:01
@vcunat:matrix.orgvcunat joined the room.18:54:02
@vcunat:matrix.orgvcunatThis is fishy. Hundreds of "queued jobs" but all have finished hours ago: https://hydra.nixos.org/eval/1739175#tabs-unfinished Fortunately, it didn't prevent the channel from advancing 😅19:00:54

Show newer messages


Back to Room ListRoom Version: 6