!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

414 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.127 Servers

Load older messages


SenderMessageTime
8 Apr 2026
@elvishjerricco:matrix.orgElvishJerricco hexa (signing key rotation when): I've pushed a new version to https://github.com/NixOS/nixpkgs/pull/435781, which doesn't include the for-WIP-only jobset file used by https://hydra.nixos.org/jobset/nixos/pr-435781-systemd-stage1. So we should disable the jobset. I don't think we need another run of it. 00:38:36
@elvishjerricco:matrix.orgElvishJerricco vcunat: ^ 00:38:41
@hexa:lossy.networkhexadisabled00:39:17
@tort55:matrix.orgtort55 joined the room.02:36:32
9 Apr 2026
@adam:robins.wtfadamcstephensI'm trying to get the revision that hydra is actively building, e.g. for nixos-unstable right now https://hydra.nixos.org/eval/1824324 . I can get this from the jobset evals API, but this is an expensive query that gathers all the build IDs as well, and can take over a minute to return a bunch of data that I don't want. Is there a more lightweight way to retrieve this info?13:47:27
10 Apr 2026
@tbutter:matrix.orgtbutter joined the room.20:07:13
11 Apr 2026
@hexa:lossy.networkhexawe run out of disk space on mimas01:20:51
@hexa:lossy.networkhexagc is running but it'll take a while to meainingfully free up space01:21:37
@hexa:lossy.networkhexa
NAME                USED  AVAIL  REFER  MOUNTPOINT
zroot              1.66T  21.7G    96K  none
01:22:23
@hexa:lossy.networkhexaI'm going to sleep, I have a script running that restarts things once we have 200G avail disk space01:22:44
@hexa:lossy.networkhexa💤01:23:15
@vcunat:matrix.orgVladimír ČunátIt works now but not much free space. I wonder why there was such a sharp spike in usage. It looks like hundreds of gigabytes within a couple hours.04:46:57
@vcunat:matrix.orgVladimír ČunátAh no. The queue runner wasn't started, so we were building nothing. Starting now.04:52:39
@vcunat:matrix.orgVladimír Čunát* Ah no. The queue runner wasn't started, so we were building nothing. Starting now. (also evaluator, but that one wasn't so important)04:53:25
@arianvp:matrix.orgArian @winter:catgirl.cloud: @winterqt:nixos.dev: i have a time-pressing issue. We enrolled the apple Macs with the MDM Push certificate you created and uploaded. We must renew them within the next 4 days. If we fail to do so the Macs turn into bricks unless we manually unenroll them now. Could you please today log into Mosyle and follow the instructions with renewing? Or can you share in the infra vault the credentials for winter@nixos.org managed apple id? 08:10:27
@arianvp:matrix.orgArianApparently the MDM Push cert is hard-bound to the appleid they were created with. So only you can renew them ._.08:10:54
@arianvp:matrix.orgArianWell they won't brick I think but we'd have to get people to manually re-enroll them most likely. But id prefer not to have to do that 08:16:34
@arianvp:matrix.orgArianI can also reset the password and 2fa in apple Business Manager myself as another option08:18:13
@hexa:lossy.networkhexawe're also keeping around 620G as nar-cache11:54:50
@hexa:lossy.networkhexanot sure how reasonable that is11:54:58
@vcunat:matrix.orgVladimír ČunátThat doesn't sound reasonable to me.11:59:51
@hexa:lossy.networkhexaand 120 GB build logs12:00:31
@vcunat:matrix.orgVladimír ČunátBuild logs get dumped to S3, so I don't expect we need a big cache here.12:06:26
@hexa:lossy.networkhexawe keep 3 years currently12:07:41
@hexa:lossy.networkhexareducing it to 7 months … aka one release cycle is probably fine12:08:18
@hexa:lossy.networkhexa
[Tue Apr  7 07:11:07 2026] AMD-Vi: Using global IVHD EFR:0x25bf732fa2295afe, EFR2:0x1d
[Tue Apr  7 07:11:07 2026] pci 0000:c0:00.2: AMD-Vi: IOMMU performance counters supported
[Tue Apr  7 07:11:07 2026] pci 0000:80:00.2: AMD-Vi: IOMMU performance counters supported
[Tue Apr  7 07:11:07 2026] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[Tue Apr  7 07:11:07 2026] pci 0000:40:00.2: AMD-Vi: IOMMU performance counters supported
[Tue Apr  7 07:11:07 2026] AMD-Vi: Extended features (0x25bf732fa2295afe, 0x1d): PPR X2APIC NX GT [5] IA GA PC GA_vAPIC
[Tue Apr  7 07:11:07 2026] AMD-Vi: Interrupt remapping enabled
[Tue Apr  7 07:11:07 2026] AMD-Vi: X2APIC enabled
[Tue Apr  7 07:11:07 2026] AMD-Vi: Virtual APIC enabled
[Tue Apr  7 07:28:55 2026] AMD-Vi: Completion-Wait loop timed out
[Tue Apr  7 12:06:39 2026] AMD-Vi: Completion-Wait loop timed out
[Tue Apr  7 14:33:26 2026] AMD-Vi: Completion-Wait loop timed out
[Tue Apr  7 14:34:09 2026] AMD-Vi: Completion-Wait loop timed out
[Tue Apr  7 15:36:00 2026] AMD-Vi: Completion-Wait loop timed out
[Tue Apr  7 18:11:58 2026] AMD-Vi: Completion-Wait loop timed out
[Tue Apr  7 18:41:58 2026] AMD-Vi: Completion-Wait loop timed out
[Wed Apr  8 00:39:41 2026] AMD-Vi: Completion-Wait loop timed out
[Wed Apr  8 01:42:46 2026] AMD-Vi: Completion-Wait loop timed out
[Wed Apr  8 02:30:06 2026] AMD-Vi: Completion-Wait loop timed out
[Wed Apr  8 06:51:02 2026] AMD-Vi: Completion-Wait loop timed out
[Wed Apr  8 06:55:58 2026] AMD-Vi: Completion-Wait loop timed out
[Wed Apr  8 12:07:31 2026] AMD-Vi: Completion-Wait loop timed out
[Wed Apr  8 13:03:08 2026] AMD-Vi: Completion-Wait loop timed out
[Wed Apr  8 14:06:48 2026] AMD-Vi: Completion-Wait loop timed out
[Wed Apr  8 14:30:59 2026] AMD-Vi: Completion-Wait loop timed out
[Wed Apr  8 15:52:20 2026] AMD-Vi: Completion-Wait loop timed out
[Wed Apr  8 15:54:29 2026] AMD-Vi: Completion-Wait loop timed out
[Wed Apr  8 20:12:38 2026] AMD-Vi: Completion-Wait loop timed out
[Wed Apr  8 21:17:51 2026] AMD-Vi: Completion-Wait loop timed out
[Wed Apr  8 21:44:14 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 00:19:30 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 00:59:30 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 01:01:57 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 03:28:53 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 07:08:24 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 08:02:55 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 09:10:49 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 10:14:00 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 10:50:43 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 13:21:57 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 13:41:20 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 14:05:22 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 14:53:35 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 15:45:57 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 16:15:29 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 16:16:06 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 16:59:11 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 17:02:36 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 19:24:10 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 23:00:36 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 23:26:18 2026] AMD-Vi: Completion-Wait loop timed out
[Thu Apr  9 23:33:49 2026] AMD-Vi: Completion-Wait loop timed out
[Fri Apr 10 02:15:21 2026] AMD-Vi: Completion-Wait loop timed out
[Fri Apr 10 02:32:24 2026] AMD-Vi: Completion-Wait loop timed out
[Fri Apr 10 03:35:33 2026] AMD-Vi: Completion-Wait loop timed out
[Fri Apr 10 06:41:22 2026] AMD-Vi: Completion-Wait loop timed out
[Fri Apr 10 06:47:59 2026] AMD-Vi: Completion-Wait loop timed out
[Fri Apr 10 07:33:30 2026] AMD-Vi: Completion-Wait loop timed out
[Fri Apr 10 08:14:39 2026] AMD-Vi: Completion-Wait loop timed out
[Fri Apr 10 08:55:03 2026] AMD-Vi: Completion-Wait loop timed out
[Fri Apr 10 09:20:51 2026] AMD-Vi: Completion-Wait loop timed out
[Fri Apr 10 09:56:58 2026] AMD-Vi: Completion-Wait loop timed out
[Fri Apr 10 11:30:25 2026] AMD-Vi: Completion-Wait loop timed out
[Fri Apr 10 12:01:33 2026] AMD-Vi: Completion-Wait loop timed out
[Fri Apr 10 12:12:48 2026] AMD-Vi: Completion-Wait loop timed out
[Fri Apr 10 16:56:16 2026] AMD-Vi: Completion-Wait loop timed out
[Fri Apr 10 18:52:19 2026] AMD-Vi: Completion-Wait loop timed out
[Fri Apr 10 20:29:55 2026] AMD-Vi: Completion-Wait loop timed out
[Sat Apr 11 00:15:14 2026] AMD-Vi: Completion-Wait loop timed out
[Sat Apr 11 01:42:46 2026] AMD-Vi: Completion-Wait loop timed out
[Sat Apr 11 04:44:27 2026] AMD-Vi: Completion-Wait loop timed out
[Sat Apr 11 11:31:24 2026] AMD-Vi: Completion-Wait loop timed out
[Sat Apr 11 11:33:50 2026] AMD-Vi: Completion-Wait loop timed out
12:09:37
@hexa:lossy.networkhexasomething is up with the IOMMU12:09:43
@hexa:lossy.networkhexaRedacted or Malformed Event12:10:12
@hexa:lossy.networkhexawell, also visible on the epyc builders12:10:29
@jfly:matrix.orgJeremy Fleischman (jfly)how would we notice if this zrepl job started failing? https://github.com/NixOS/infra/blob/main/build/titan/zrepl.nix. i can't find any monitoring/alerting.17:53:47

Show newer messages


Back to Room ListRoom Version: 6