In reply to @asymmetric:matrix.dapp.org.uk
could we add another command to ofborg, so that one could do @ofborg set timeout 2h polkadot aarch64-darwin or something, on a case by case basis, as a github comment on the pr of a specific package? this would at least be a stopgap, and would mean we don't have to "pollute" meta with ofborg-specific attributes

In reply to @cole-h:matrix.org
An interesting idea, but I think the problem with that is that 1) the timeout is set for the nix-build command (so there's no way to change it once the build has started running); and 2) the machine that processes comment commands is separate from the machines that run those commands (builds, evals, etc)...

yeah i thought that we could either:

1. add the comment in the PR body, so that it gets picked up before the job starts; or
2. add the behaviour that when this comment is set, a build is restarted; or
3. add a restart command which must also be used when using set timeout

i'm not sure i understand the implications of your point 2) though -- couldn't the comments-listening machine direct the building-machine?

In reply to @cole-h:matrix.org
My point with #2 is that (as far as I know), the coordinator doesn't know which machine took the job, so how should it determine which build machine to inform about the change? (I think the simplest depiction of how it all works, from my knowledge, is GitHub webhook -> ofborg-core (coordinator) -> AMQP server <-> ofborg-eval-X (evaluator and builder))
But the "comment in PR body" could work, if ofborg is notified about that information (it must be, in some roundabout way, because we have access to the PR number)......

In reply to @cole-h:matrix.org
My point with #2 is that (as far as I know), the coordinator doesn't know which machine took the job, so how should it determine which build machine to inform about the change? (I think the simplest depiction of how it all works, from my knowledge, is GitHub webhook -> ofborg-core (coordinator) -> AMQP server <-> ofborg-eval-X (evaluator and builder))
But the "comment in PR body" could work, if ofborg is notified about that information (it must be, in some roundabout way, because we have access to the PR number)......

In reply to @hexa:lossy.network
don't think we monitor the darwin machines in prometheus

In reply to @cafkafk:gitter.im
disk full https://github.com/NixOS/nixpkgs/pull/258395/checks?check_run_id=17292775692

curl -ss http://...:9100/metrics | head
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.9458e-05
go_gc_duration_seconds{quantile="0.25"} 5.4583e-05
go_gc_duration_seconds{quantile="0.5"} 6.6707e-05
go_gc_duration_seconds{quantile="0.75"} 9.1292e-05
go_gc_duration_seconds{quantile="1"} 0.000423
go_gc_duration_seconds_sum 251.971410887
go_gc_duration_seconds_count 115795
# HELP go_goroutines Number of goroutines that currently exist.