| 10 Jun 2024 |
|  mjolnir unbanned  @jonringer:matrix.org. | 00:17:03 |
| 13 Jun 2024 |
 shawn8901 | Hydra seems to be dead due to not being able to connect to the database 🫣 | 06:54:36 |
| shawn8901 | In reply to @shawn8901:matrix.org
Hydra seems to be dead due to not being able to connect to the database 🫣 Seems to be up again | 07:05:08 |
 vcunat | Yes, I'm watching it. We still haven't really resolved the full-disk issues. | 07:10:39 |
| vcunat | Anyway, this channel is about the software. Hydra.nixos.org falls under #infra:nixos.org | 07:10:58 |
| 16 Jun 2024 |
|  Pyrox [Fox/It/She] joined the room. | 01:22:18 |
| 18 Jun 2024 |
| shawn8901 | Is there some way with hydra to handle part-time build machines? I want to add my desktop PC as additional remote builder tho it's not running all the time, I tried that in the past but then hydra was crying about the remote builder not being alive.
I tried to look at docs but did not find anything in that direction. | 12:11:36 |
| shawn8901 | * Is there some way with hydra to handle part-time build machines? I want to add my desktop PC as additional remote builder as it's much more powerful, tho it's not running all the time, I tried that in the past but then hydra was crying about the remote builder not being alive.
I tried to look at docs but did not find anything in that direction. | 12:12:02 |
| shawn8901 | * Is there some way with hydra to handle part-time build machines? I want to add my desktop PC as additional remote builder as it's much more powerful, tho it's not running all the time, I tried that in the past but then hydra was failing jobs as the remote builder not being alive.
I tried to look at docs but did not find anything in that direction. | 12:14:55 |
 cransom | I had a script that would discover alive machines and then add them to the build machines file. | 12:26:38 |
| shawn8901 | So the buildMachine file is read before enqueing? | 13:14:51 |
| shawn8901 | Ah okay it seems that there is a watcher. Hum maybe that's an idea, I could add it on boot and remove on shutdown or so | 13:18:22 |
| cransom | yeah, it's re-read before job distribution iirc. I used it to autoscale builders by checking for machines in an autoscaling group in AWS. | 14:23:49 |
| vcunat | Yes, modifying the file takes effect immediately. (for any newly built steps) | 14:40:07 |
| 20 Jun 2024 |
 hexa | I have an issue with hydra where it tells me builders have possibly transient build failures | 11:59:39 |
| hexa | 
Download image.png | 11:59:41 |
| hexa | that leads to these 93 jobs not being built | 11:59:53 |
| hexa | I can push changes to the tracked branch and the diff of jobs will be built, but these 93 jobs won't | 12:00:18 |
| hexa | hydra is not really very verbose, what the issue at hand is | 12:00:35 |
| hexa | hydra-queue-runner[301701]: possibly transient failure building ‘/nix/store/ksrvihi2zlm20qaqvxq0ryc4kkhyxjf8-grpcio-tools-1.64.1.tar.gz.drv’ on ‘hexa@build1.darmstadt.ccc.de’:
hydra-queue-runner[301701]: will retry ‘/nix/store/ksrvihi2zlm20qaqvxq0ryc4kkhyxjf8-grpcio-tools-1.64.1.tar.gz.drv’ after 66s
| 12:00:52 |
| hexa | example | 12:00:54 |
| hexa | * as an example\ | 12:00:59 |
| hexa | * as an example | 12:01:01 |
| hexa | * I can push changes to the tracked branch and evaluate them, which results in the changed jobs being built, but these 93 jobs simply won't | 12:01:33 |
| hexa | hm, these are all fods which have broken urls | 12:04:24 |
| vcunat | FODs are hard. Normally hydra won't rebuild them unless the output hash changes. | 12:18:00 |
| vcunat | But just changing URL won't change that. | 12:18:12 |
| vcunat | * FODs are hard. Normally hydra won't create a new job unless the output hash changes. | 12:18:49 |
| vcunat | So you're stuck with the old job tied to the incorrect .drv | 12:19:03 |
| vcunat | As an ugly workaround, you could evaluate in a _different) jobset, for example. That will get a new .drv (if you evaluate such commits right away), getting you eventually binaries and then you can just restart these failed jobs to get the cached results and you'll get green. | 12:20:27 |
