please adopt nixos/infra gc settings on the ofborg macs, so monitoring shuts up 🙂

    gc = {
      automatic = true;
      interval = {
        # hourly at the 15th minute
        Minute = 15;
      };
      # ensure up to 125G free space every hour
      options = "--max-freed $(df -k /nix/store | awk 'NR==2 {available=$4; required=125*1024*1024; to_free=required-available; printf \"%.0d\", to_free*1024}')";
    };

Looking at the Grafana dashboard for the past two days, it looks like builders 01, 02 and 04 stopped building things at around 10:00 UTC yesterday and only started building again ~3 hours ago, which somewhat lines up with when GitHub PRs stopped having ofBorg build reports and when they started having them again. I don't think we have more public data sources (logs, more metrics, etc) to figure out why this happened though, so that's the extend of the help I can be.

There was also a GH outage a couple hours later that impacted webhooks, so it might be related if this started earlier and they failed to report it? https://www.githubstatus.com/incidents/6swp0zf7lk8h

PS. builder03 seems to have stopped building since August 1st?

I am seeing the below in my PR, and it seems to be happening in quite a few others:

       … while evaluating attribute 'args' of derivation 'bootstrap-stage0-glibc-bootstrapFiles'
         at /var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ofborg-core/pkgs/stdenv/generic/make-derivation.nix:510:13:
          509|             builder = attrs.realBuilder or stdenv.shell;
          510|             args =
             |             ^
          511|               attrs.args or [

       error: cannot open connection to remote store 'daemon': error: reading from file: Connection reset by peer