| 22 Jun 2026 |
hexa | yeah, I think we're running into that 4 jobs limit a lot | 11:44:25 |
Mic92 | Maybe but we also regressed in performance when I had to go back to using the nix-daemon for more stuff. I try to get some of the performance back but as long as things are somewhat stable now, I would like to move a bit slower and do more testing | 11:48:39 |
Vladimír Čunát | Did buildMozillaMach already suffer from silent-timeouts during the final link? | 12:21:02 |
hexa | you mean before today? I'd say no | 12:21:22 |
Vladimír Čunát | e.g. this 3h build got it twice already:
https://hydra.nixos.org/build/331659622#tabs-buildsteps | 12:21:36 |
hexa | silent-timeout on linux should be 3h | 12:22:06 |
hexa | so let me just say that 2h 59m 2s is under 3h | 12:22:23 |
Vladimír Čunát | Maybe we're not applying it correctly with the new queue-runner anymore? 🤔 | 12:22:37 |
Vladimír Čunát | * Maybe we're not applying such settings correctly with the new queue-runner anymore? 🤔 | 12:23:01 |
hexa | looks like it defaults to 3600s | 12:23:38 |
hexa | so, one hour of linking … not unheard of in a competing builds scenario | 12:24:11 |
hexa | #[tracing::instrument(skip(v, jobset), err)]
pub fn new(v: db::models::Build, jobset: Arc<Jobset>) -> Result<Arc<Self>, jiff::Error> {
Ok(Arc::new(Self {
id: v.id,
drv_path: v.drvpath,
outputs: BTreeMap::new(),
jobset_id: v.jobset_id,
name: v.job,
timestamp: jiff::Timestamp::from_second(v.timestamp)?,
max_silent_time: v.maxsilent.unwrap_or(3600),
timeout: v.timeout.unwrap_or(36000),
local_priority: v.priority,
global_priority: v.globalpriority.into(),
toplevel: arc_swap::ArcSwapOption::from(None),
jobset,
finished_in_db: false.into(),
}))
}
| 12:24:30 |
hexa | Redacted or Malformed Event | 12:24:45 |
Mic92 | hexa (signing key rotation when): https://github.com/NixOS/infra/pull/1098 https://github.com/NixOS/infra/pull/1099 both should be go through staging hydra, but I won't have time for this just now | 12:24:47 |
hexa | it's fine, let's take the perf stuff slow and the regression fixes fast | 12:25:42 |
hexa | Redacted or Malformed Event | 12:26:17 |
Vladimír Čunát |
in a competing builds scenario
There was almost no CPU pressure during the last attempt on that builder.
| 12:27:04 |
hexa | the lack of timestamps in build logs … sigh | 12:27:21 |
Vladimír Čunát | But it's possible. The links of FF/TB have been really long lately, and aarch64-linux probably has lower single-thread performance. | 12:27:51 |
hexa | yeah, I mean, even for my builds the link takes 30 minutes or so on firefox | 12:28:14 |
hexa | so I wouldn't be surprised about an hour of silence, if that goes slower | 12:28:25 |
Mic92 |
283: maxSilent = 14400; # 4h, double the default of 7200s; sometimes needed for x86_64-darwin on hydra
| 12:28:48 |
Mic92 | maybe one more candidate where it's not read correctly from the derivation? | 12:29:04 |
hexa | and from nix.conf | 12:29:25 |
Mic92 | structured attrs would the first thing to check | 12:30:05 |
hexa | we configure nix.settings.max-silent-time and it just hardcodes 3600 instead | 12:30:33 |
Mic92 | https://github.com/NixOS/hydra/commits/hydra.nixos.org/ made this and two other limits configureable | 12:44:10 |
Mic92 | The performance changes are now in staging-hydra.nixos.org branch | 12:44:28 |
Mic92 | https://github.com/NixOS/hydra/commits/staging-hydra.nixos.org | 12:44:37 |
hexa | hm, I saw two chromium builds complete after 1d 1h yesterday | 12:45:46 |