!sBfrWMVsLoSyFTCkNv:nixos.org

OfBorg

180 Members
Number of builds and evals in queue: https://ofborg.org/prometheus/graph?g0.expr=ofborg_queue_evaluator_waiting&g0.tab=1&g0.stacked=0&g0.show_exemplars=0&g0.range_input=2h&g1.expr=ofborg_queue_builder_waiting%7Barch!~%22.*-lowprior%22%7D&g1.tab=1&g1.stacked=0&g1.show_exemplars=0&g1.range_input=2h63 Servers

Load older messages


SenderMessageTime
29 Sep 2023
@asymmetric:matrix.dapp.org.uk@asymmetric:matrix.dapp.org.uki wonder if alternatively we could tell ofborg not to build the package on certain archs at all. rn it's just wasting precious cycles09:45:03
@asymmetric:matrix.dapp.org.uk@asymmetric:matrix.dapp.org.uk * i wonder if alternatively we could tell ofborg not to build the package on certain arches at all. rn it's just wasting precious cycles09:45:12
@raitobezarius:matrix.orgraitobezariusbadPlatforms ?12:57:56
@7c6f434c:nitro.chat7c6f434cThe question seems to be distinguishing «broken there», «don't build on Hydra for resource reasons including space» and «will timeout on ofBorg»12:59:37
@lily:lily.flowersLily FosterYeah there's a meta attribute for controlling hydra platforms, but there is not for ofborg currently13:17:50
@lily:lily.flowersLily Foster I mean I think ofborg does pure eval, so you could try to detect it and those bad platforms in meta.broken 🙈 14:02:21
@lily:lily.flowersLily FosterRedacted or Malformed Event14:02:25
@lily:lily.flowersLily Foster* I mean I think ofborg does impure eval, so you could try to detect it and those bad platforms in meta.broken 🙈14:02:31
@lily:lily.flowersLily Foster* I mean I think ofborg does impure eval, so you could try to detect it and those bad platforms in `meta.broken` 🙈14:02:44
@lily:lily.flowersLily Foster
In reply to @lily:lily.flowers
I mean I think ofborg does impure eval, so you could try to detect it and those bad platforms in meta.broken 🙈
(Pls don't actually do that, I will cry if you do)
14:03:56
@7c6f434c:nitro.chat7c6f434cI don't think ofBorg and normal impure eval are distinguishable, and we don't want to mark the package broken for normal use.14:07:11
@lily:lily.flowersLily Foster
In reply to @7c6f434c:nitro.chat
I don't think ofBorg and normal impure eval are distinguishable, and we don't want to mark the package broken for normal use.
Well if it's impure, you could use builtins.readFile and detect hostnames or the ofborg runner executable and such 😉
14:30:18
@7c6f434c:nitro.chat7c6f434cI wonder if this will be an error-out during pure eval, though14:33:22
@lily:lily.flowersLily Foster(But again please never do that, everything about it is so cursed and frail 😭)14:33:25
@lily:lily.flowersLily Foster
In reply to @7c6f434c:nitro.chat
I wonder if this will be an error-out during pure eval, though
lib.isPureEval
14:33:37
@lily:lily.flowersLily Foster* `lib.inPureEvalMode`14:34:49
@7c6f434c:nitro.chat7c6f434cFortunately existence of stable branches would make tying to details of ofBorg deployments just too painful and impractical…14:38:34
@lily:lily.flowersLily FosterStables branches or not, it's a bad and frail idea14:41:41
@lily:lily.flowersLily FosterPlease don't seriously consider this in any capacity....14:42:19
@cole-h:matrix.orgcole-hMy problem with increasing the ofborg timeout on darwin is that the darwin builders are pretty slow as it is. I worry that that would cause the darwin queue to blow up (as has happened in the past even without a longer timeout). It would be interesting to explore an "ofborgWillTimeoutOnTheseSystems" predicate, though.15:40:09
@infinisil:matrix.org@infinisil:matrix.org cole-h: What about a dynamic approach: When the queue is too long, time out the longest-running job until it's short enough again 15:58:15
@asymmetric:matrix.dapp.org.uk@asymmetric:matrix.dapp.org.uk could we add a command to ofborg, so that one could do @ofborg set timeout 2h or something, on a case by case basis, as a github comment? this would at least be a stopgap, and would mean we don't have to "pollute" meta with ofborg-specific attributes 15:59:46
@asymmetric:matrix.dapp.org.uk@asymmetric:matrix.dapp.org.uk * could we add another command to ofborg, so that one could do @ofborg set timeout 2h or something, on a case by case basis, as a github comment? this would at least be a stopgap, and would mean we don't have to "pollute" meta with ofborg-specific attributes 15:59:54
@asymmetric:matrix.dapp.org.uk@asymmetric:matrix.dapp.org.uk * could we add another command to ofborg, so that one could do @ofborg set timeout 2h aarch64-darwin or something, on a case by case basis, as a github comment? this would at least be a stopgap, and would mean we don't have to "pollute" meta with ofborg-specific attributes 16:00:06
@asymmetric:matrix.dapp.org.uk@asymmetric:matrix.dapp.org.uk * could we add another command to ofborg, so that one could do @ofborg set timeout 2h aarch64-darwin or something, on a case by case basis, as a github comment on the pr of a specific package? this would at least be a stopgap, and would mean we don't have to "pollute" meta with ofborg-specific attributes 16:00:24
@asymmetric:matrix.dapp.org.uk@asymmetric:matrix.dapp.org.uk * could we add another command to ofborg, so that one could do @ofborg set timeout 2h polkadot aarch64-darwin or something, on a case by case basis, as a github comment on the pr of a specific package? this would at least be a stopgap, and would mean we don't have to "pollute" meta with ofborg-specific attributes 16:00:31
@cole-h:matrix.orgcole-h
In reply to @infinisil:matrix.org
cole-h: What about a dynamic approach: When the queue is too long, time out the longest-running job until it's short enough again
As far as I know, RabbitMQ (what ofborg uses) is a "dumb" queue system. I don't know if we get information about the queue aside from the fact that there's a job we can take, and then communicating that we succeeded a job...
(I'm not all that familiar with RabbitMQ, however)
16:01:32
@cole-h:matrix.orgcole-h
In reply to @asymmetric:matrix.dapp.org.uk
could we add another command to ofborg, so that one could do @ofborg set timeout 2h polkadot aarch64-darwin or something, on a case by case basis, as a github comment on the pr of a specific package? this would at least be a stopgap, and would mean we don't have to "pollute" meta with ofborg-specific attributes
An interesting idea, but I think the problem with that is that 1) the timeout is set in the nix-build command (so there's no way to change it once the build has started running); and 2) the machine that processes comment commands is separate from the machines that run those commands (builds, evals, etc)...
16:03:22
@cole-h:matrix.orgcole-h
In reply to @asymmetric:matrix.dapp.org.uk
could we add another command to ofborg, so that one could do @ofborg set timeout 2h polkadot aarch64-darwin or something, on a case by case basis, as a github comment on the pr of a specific package? this would at least be a stopgap, and would mean we don't have to "pollute" meta with ofborg-specific attributes
* An interesting idea, but I think the problem with that is that 1) the timeout is set for the nix-build command (so there's no way to change it once the build has started running); and 2) the machine that processes comment commands is separate from the machines that run those commands (builds, evals, etc)...
16:03:38
@asymmetric:matrix.dapp.org.uk@asymmetric:matrix.dapp.org.uk
In reply to @cole-h:matrix.org
An interesting idea, but I think the problem with that is that 1) the timeout is set for the nix-build command (so there's no way to change it once the build has started running); and 2) the machine that processes comment commands is separate from the machines that run those commands (builds, evals, etc)...

yeah i thought that we could either:

  1. add the comment in the PR body, so that it gets picked up before the job starts; or
  2. add the behaviour that when this comment is set, a build is restarted; or
  3. add a restart command which must also be used when using set timeout

i'm not sure i understand the implications of your point 2) though -- couldn't the comments-listening machine direct the building-machine?

16:06:12

Show newer messages


Back to Room ListRoom Version: 6