!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

476 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.150 Servers

Load older messages


SenderMessageTime
22 Jun 2026
@hexa:lossy.networkhexayeah, I think we're running into that 4 jobs limit a lot11:44:25
@joerg:thalheim.ioMic92Maybe but we also regressed in performance when I had to go back to using the nix-daemon for more stuff. I try to get some of the performance back but as long as things are somewhat stable now, I would like to move a bit slower and do more testing11:48:39
@vcunat:matrix.orgVladimír ČunátDid buildMozillaMach already suffer from silent-timeouts during the final link?12:21:02
@hexa:lossy.networkhexayou mean before today? I'd say no12:21:22
@vcunat:matrix.orgVladimír Čunáte.g. this 3h build got it twice already: https://hydra.nixos.org/build/331659622#tabs-buildsteps12:21:36
@hexa:lossy.networkhexasilent-timeout on linux should be 3h12:22:06
@hexa:lossy.networkhexaso let me just say that 2h 59m 2s is under 3h12:22:23
@vcunat:matrix.orgVladimír ČunátMaybe we're not applying it correctly with the new queue-runner anymore? 🤔12:22:37
@vcunat:matrix.orgVladimír Čunát* Maybe we're not applying such settings correctly with the new queue-runner anymore? 🤔12:23:01
@hexa:lossy.networkhexalooks like it defaults to 3600s12:23:38
@hexa:lossy.networkhexaso, one hour of linking … not unheard of in a competing builds scenario12:24:11
@hexa:lossy.networkhexa
    #[tracing::instrument(skip(v, jobset), err)]
    pub fn new(v: db::models::Build, jobset: Arc<Jobset>) -> Result<Arc<Self>, jiff::Error> {
        Ok(Arc::new(Self {
            id: v.id,
            drv_path: v.drvpath,
            outputs: BTreeMap::new(),
            jobset_id: v.jobset_id,
            name: v.job,
            timestamp: jiff::Timestamp::from_second(v.timestamp)?,
            max_silent_time: v.maxsilent.unwrap_or(3600),
            timeout: v.timeout.unwrap_or(36000),
            local_priority: v.priority,
            global_priority: v.globalpriority.into(),
            toplevel: arc_swap::ArcSwapOption::from(None),
            jobset,
            finished_in_db: false.into(),
        }))
    }
12:24:30
@hexa:lossy.networkhexaRedacted or Malformed Event12:24:45
@joerg:thalheim.ioMic92 hexa (signing key rotation when): https://github.com/NixOS/infra/pull/1098 https://github.com/NixOS/infra/pull/1099 both should be go through staging hydra, but I won't have time for this just now 12:24:47
@hexa:lossy.networkhexait's fine, let's take the perf stuff slow and the regression fixes fast12:25:42
@hexa:lossy.networkhexaRedacted or Malformed Event12:26:17
@vcunat:matrix.orgVladimír Čunát

in a competing builds scenario

There was almost no CPU pressure during the last attempt on that builder.

12:27:04
@hexa:lossy.networkhexathe lack of timestamps in build logs … sigh12:27:21
@vcunat:matrix.orgVladimír ČunátBut it's possible. The links of FF/TB have been really long lately, and aarch64-linux probably has lower single-thread performance.12:27:51
@hexa:lossy.networkhexayeah, I mean, even for my builds the link takes 30 minutes or so on firefox12:28:14
@hexa:lossy.networkhexaso I wouldn't be surprised about an hour of silence, if that goes slower12:28:25
@joerg:thalheim.ioMic92

283: maxSilent = 14400; # 4h, double the default of 7200s; sometimes needed for x86_64-darwin on hydra

12:28:48
@joerg:thalheim.ioMic92maybe one more candidate where it's not read correctly from the derivation?12:29:04
@hexa:lossy.networkhexaand from nix.conf12:29:25
@joerg:thalheim.ioMic92structured attrs would the first thing to check12:30:05
@hexa:lossy.networkhexa we configure nix.settings.max-silent-time and it just hardcodes 3600 instead 12:30:33
@joerg:thalheim.ioMic92https://github.com/NixOS/hydra/commits/hydra.nixos.org/ made this and two other limits configureable12:44:10
@joerg:thalheim.ioMic92The performance changes are now in staging-hydra.nixos.org branch12:44:28
@joerg:thalheim.ioMic92https://github.com/NixOS/hydra/commits/staging-hydra.nixos.org12:44:37
@hexa:lossy.networkhexahm, I saw two chromium builds complete after 1d 1h yesterday12:45:46

Show newer messages


Back to Room ListRoom Version: 6