!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

483 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.151 Servers

Load older messages


SenderMessageTime
22 Jun 2026
@hexa:lossy.networkhexaRedacted or Malformed Event11:15:54
@hexa:lossy.networkhexa I checked store dir and build dir is available (>20%) and pressure is normal 11:16:28
@hexa:lossy.networkhexathe only remaining check I can't evaluate in my head is11:16:47
@hexa:lossy.networkhexa
        let now = jiff::Timestamp::now().as_second();
        let jobs_in_last_30s_start = self.stats.jobs_in_last_30s_start.load(Ordering::Relaxed);
        let jobs_in_last_30s_count = self.stats.jobs_in_last_30s_count.load(Ordering::Relaxed);

        // ensure that we dont submit more than 4 jobs in 30s
        if now <= (jobs_in_last_30s_start + 30)
            && jobs_in_last_30s_count >= 4
            // ensure that we havent already finished some of them, because then its fine again
            && self.stats.get_current_jobs() >= 4
        {
            return false;
        } else if now > (jobs_in_last_30s_start + 30) {
            // reset count
            self.stats
                .jobs_in_last_30s_start
                .store(0, Ordering::Relaxed);
            self.stats
                .jobs_in_last_30s_count
                .store(0, Ordering::Relaxed);
        }

11:16:51
@hexa:lossy.networkhexaRedacted or Malformed Event11:16:53
@hexa:lossy.networkhexaRedacted or Malformed Event11:16:57
@hexa:lossy.networkhexahm, ok. now it's back to building packages and it runs more jobs, previously it was busy with nixos tests11:18:12
@hexa:lossy.networkhexaalso the queue has a new eval11:18:42
@hexa:lossy.networkhexayeah, I think we're running into that 4 jobs limit a lot11:44:25
@joerg:thalheim.ioMic92Maybe but we also regressed in performance when I had to go back to using the nix-daemon for more stuff. I try to get some of the performance back but as long as things are somewhat stable now, I would like to move a bit slower and do more testing11:48:39
@vcunat:matrix.orgVladimír ČunátDid buildMozillaMach already suffer from silent-timeouts during the final link?12:21:02
@hexa:lossy.networkhexayou mean before today? I'd say no12:21:22
@vcunat:matrix.orgVladimír Čunáte.g. this 3h build got it twice already: https://hydra.nixos.org/build/331659622#tabs-buildsteps12:21:36
@hexa:lossy.networkhexasilent-timeout on linux should be 3h12:22:06
@hexa:lossy.networkhexaso let me just say that 2h 59m 2s is under 3h12:22:23
@vcunat:matrix.orgVladimír ČunátMaybe we're not applying it correctly with the new queue-runner anymore? 🤔12:22:37
@vcunat:matrix.orgVladimír Čunát* Maybe we're not applying such settings correctly with the new queue-runner anymore? 🤔12:23:01
@hexa:lossy.networkhexalooks like it defaults to 3600s12:23:38
@hexa:lossy.networkhexaso, one hour of linking … not unheard of in a competing builds scenario12:24:11
@hexa:lossy.networkhexa
    #[tracing::instrument(skip(v, jobset), err)]
    pub fn new(v: db::models::Build, jobset: Arc<Jobset>) -> Result<Arc<Self>, jiff::Error> {
        Ok(Arc::new(Self {
            id: v.id,
            drv_path: v.drvpath,
            outputs: BTreeMap::new(),
            jobset_id: v.jobset_id,
            name: v.job,
            timestamp: jiff::Timestamp::from_second(v.timestamp)?,
            max_silent_time: v.maxsilent.unwrap_or(3600),
            timeout: v.timeout.unwrap_or(36000),
            local_priority: v.priority,
            global_priority: v.globalpriority.into(),
            toplevel: arc_swap::ArcSwapOption::from(None),
            jobset,
            finished_in_db: false.into(),
        }))
    }
12:24:30
@hexa:lossy.networkhexaRedacted or Malformed Event12:24:45
@joerg:thalheim.ioMic92 hexa (signing key rotation when): https://github.com/NixOS/infra/pull/1098 https://github.com/NixOS/infra/pull/1099 both should be go through staging hydra, but I won't have time for this just now 12:24:47
@hexa:lossy.networkhexait's fine, let's take the perf stuff slow and the regression fixes fast12:25:42
@hexa:lossy.networkhexaRedacted or Malformed Event12:26:17
@vcunat:matrix.orgVladimír Čunát

in a competing builds scenario

There was almost no CPU pressure during the last attempt on that builder.

12:27:04
@hexa:lossy.networkhexathe lack of timestamps in build logs … sigh12:27:21
@vcunat:matrix.orgVladimír ČunátBut it's possible. The links of FF/TB have been really long lately, and aarch64-linux probably has lower single-thread performance.12:27:51
@hexa:lossy.networkhexayeah, I mean, even for my builds the link takes 30 minutes or so on firefox12:28:14
@hexa:lossy.networkhexaso I wouldn't be surprised about an hour of silence, if that goes slower12:28:25
@joerg:thalheim.ioMic92

283: maxSilent = 14400; # 4h, double the default of 7200s; sometimes needed for x86_64-darwin on hydra

12:28:48

Show newer messages


Back to Room ListRoom Version: 6