| 2 Aug 2025 |
hexa | [FIRING - WARNING]
PartitionLowDiskSpace: /dev/disk3s1s1 mounted to / (apfs) on mac04.ofborg.org:9100 has 7% free.
(
📈 Grafana,
🔥 Prometheus,
🔕 Silence
)
[FIRING - WARNING]
PartitionLowDiskSpace: /dev/disk/by-label/nixos mounted to / (ext4) on tracker.security.nixos.org:9100 has 8% free.
(
📈 Grafana,
🔥 Prometheus,
🔕 Silence
)
| 16:23:21 |
hexa | I have a feeling we need to gc more aggressively on the macs | 16:23:35 |
| 6 Aug 2025 |
@drupol:matrix.org | Hey hey hellow | 14:03:53 |
@drupol:matrix.org | Do you have any idea on what is going on with the GHA CI builds ? https://discourse.nixos.org/t/no-more-ci-builds/67654 | 14:04:09 |
dgrig | Looking at the Grafana dashboard for the past two days, it looks like builders 01, 02 and 04 stopped building things at around 10:00 UTC yesterday and only started building again ~3 hours ago, which somewhat lines up with when GitHub PRs stopped having ofBorg build reports and when they started having them again. I don't think we have more public data sources (logs, more metrics, etc) to figure out why this happened though, so that's the extend of the help I can be.
There was also a GH outage a couple hours later that impacted webhooks, so it might be related if this started earlier and they failed to report it? https://www.githubstatus.com/incidents/6swp0zf7lk8h
PS. builder03 seems to have stopped building since August 1st?
| 15:36:21 |
emily | these are the donated liberodark builders? | 15:58:27 |
emily | is there any line of contact? | 15:58:33 |
hexa | they did not go offline, so rather a scheduling thing | 16:01:19 |
hexa | * they did not go offline, so rather a scheduling thing I'd imagine | 16:01:21 |
hexa | or maybe github stopped sending webhooks, who knows | 16:07:49 |
| 8 Aug 2025 |
| 0x4A6F joined the room. | 06:58:50 |
| 10 Aug 2025 |
| tris203 joined the room. | 20:32:58 |
tris203 | I am seeing the below in my PR, and it seems to be happening in quite a few others:
… while evaluating attribute 'args' of derivation 'bootstrap-stage0-glibc-bootstrapFiles'
at /var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/ofborg-core/pkgs/stdenv/generic/make-derivation.nix:510:13:
509| builder = attrs.realBuilder or stdenv.shell;
510| args =
| ^
511| attrs.args or [
error: cannot open connection to remote store 'daemon': error: reading from file: Connection reset by peer
| 20:33:27 |
| 12 Aug 2025 |
K900 | Can someone take a look at why https://github.com/NixOS/nixpkgs/pull/430298 failed | 17:58:17 |
| 16 Aug 2025 |
| adamcstephens joined the room. | 14:11:57 |
| 29 Aug 2025 |
kirillrdy | any idea why ofborg didnt build this PR https://github.com/NixOS/nixpkgs/pull/438138 ? | 08:51:47 |
Simon Hauser | fixed, comment poster crashed. restarted fixed it. builds are now show, also for this PR | 10:04:23 |
kirillrdy | thank you ❤️ | 10:06:16 |
| @drupol:matrix.org left the room. | 11:42:44 |
| 1 Sep 2025 |
Tristan Ross | https://github.com/NixOS/nixpkgs/pull/437137#issuecomment-3240658296 ofborg just seems broken | 02:24:22 |
Alyssa Ross | Redacted or Malformed Event | 13:48:58 |
hexa | https://github.com/NixOS/nixpkgs/pull/438399#issuecomment-3242957873 | 17:16:12 |
hexa | an example | 17:16:14 |
| 2 Sep 2025 |
| Wolfgang Walther joined the room. | 14:02:05 |
| winston joined the room. | 16:40:27 |
winston | https://github.com/NixOS/nixpkgs/pull/435249 got the ofborg-internal-error label, anything to do from my side? | 16:41:43 |
| @ihar.hrachyshka:matrix.org joined the room. | 16:42:09 |
hexa | no | 18:49:41 |
Simon Hauser | i debugged it this evening and i think i found the issue, will deploy a potential fix shortly | 19:43:45 |
Simon Hauser | issue is fixed, you can comment with @ofborg eval and ofborg will take another look and schedule builds for the pr | 20:19:03 |