In reply to @andreas.schraegle:helsinki-systems.de
I tried to delete a project right now and this happened:

anyone seen errors like this one before?

anyone seen errors like this one before?

possibly transient failure building ‘/nix/store/xxx-stage-1-init.sh.drv’ on ‘localhost’: dependency '/nix/store/xxx-extra-utils' of '/nix/store/xxx-stage-1-init.sh.drv' does not exist, and substitution is disabled

what I've found is it is checking to see if the destStore (an S3BinaryCacheStoreImpl) has it as a valid path:

State::createStep (this=0x7fffffff7280, destStore=..., conn=..., build=std::shared_ptr<Build> (use count 5, weak count 1) = {...}, drvPath=..., referringBuild=std::shared_ptr<Build> (empty) = {...}, 
    referringStep=std::shared_ptr<Step> (use count 2, weak count 2) = {...}, finishedDrvs=std::set with 1 element = {...}, newSteps=std::set with 0 elements, newRunnable=std::set with 0 elements)
    at queue-monitor.cc:470
470	        if (!destStore->isValidPath(*i.second.path(*localStore, step->drv->name, i.first))) {

isValidPath checks the disk cache:

   428      if (diskCache) {
   429          auto res = diskCache->lookupNarInfo(getUri(), hashPart);
   430          if (res.first != NarInfoDiskCache::oUnknown) {

it doesn't know (oUnknown) and falls through to:

   439      bool valid = isValidPathUncached(storePath);
   440  
   441      if (diskCache && !valid)
   442          // FIXME: handle valid = true case.
   443          diskCache->upsertNarInfo(getUri(), hashPart, 0);
   444  
   445      return valid;

and it does exist which ends up returning valid (true) here, with that FIXME hint.

from there we're back in the queue monitor:

   470          if (!destStore->isValidPath(*i.second.path(*localStore, step->drv->name, i.first))) {
   471              valid = false;
   472              missing.insert_or_assign(i.first, i.second);
   473          }

except the path is valid on the remote end despite not being on the local machine, so it never gets put in to "missing", so it never enters this:

   475      /* Try to copy the missing paths from the local store or from
   476         substitutes. */
   477      if (!missing.empty()) {

at which point Hydra thinks it is ready to build a derivation where a dependent derivation's outputs are not available