!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

482 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.151 Servers

Load older messages


SenderMessageTime
20 Jun 2026
@hexa:lossy.networkhexaI'm not touching anything until it is done13:17:02
@joerg:thalheim.ioMic92 hexa (signing key rotation when): looks like the accept look just exits when the number of connections exceeds a value. 13:17:46
@joerg:thalheim.ioMic92and we have 1024 FDs maximum13:17:53
@hexa:lossy.networkhexalmao13:18:14
@hexa:lossy.networkhexa
# ls /proc/3496116/fd/ | wc -l
401
13:18:52
@joerg:thalheim.ioMic92Deploying a fix now13:21:35
@joerg:thalheim.ioMic92 * hexa (signing key rotation when): looks like the accept loop just exits when the number of connections exceeds a value. 13:21:44
@joerg:thalheim.ioMic92 hexa (signing key rotation when): if you want to wait, can you deploy? 13:22:50
@hexa:lossy.networkhexasure13:22:56
@hexa:lossy.networkhexathe time for a channel update has increased significantly14:14:42
@hexa:lossy.networkhexa
May 24 14:36:44 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 10min 5.711s CPU time, 36G memory peak, 16.8M read from disk, 899.7M incoming IP traffic, 148.4M outgoing IP traffic.
May 25 13:26:30 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 10min 13.988s CPU time, 35.9G memory peak, 7.1M read from disk, 951.8M incoming IP traffic, 151.3M outgoing IP traffic.
May 26 09:17:08 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 10min 42.995s CPU time over 7min 44.995s wall clock time, 27.8G memory peak, 1.7M read from disk, 942.1M incoming IP traffic, 145.3M outgoing IP traffic.
May 28 08:04:29 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 11min 20.556s CPU time over 8min 57.699s wall clock time, 27G memory peak, 512K read from disk, 961.8M incoming IP traffic, 160.7M outgoing IP traffic.
May 29 04:12:04 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 10min 59.103s CPU time over 8min 24.238s wall clock time, 27.5G memory peak, 24K read from disk, 933.7M incoming IP traffic, 165.4M outgoing IP traffic.
May 31 06:48:12 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 10min 45.064s CPU time over 7min 47.066s wall clock time, 27.3G memory peak, 94.1M memory swap peak, 1017.9M read from disk, 91.3M written to disk, 957.5M incoming IP traffic, 155.2M outgoing IP traffic.
Jun 01 09:30:05 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 10min 41.612s CPU time over 8min 6.600s wall clock time, 27.4G memory peak, 15.6M read from disk, 954.9M incoming IP traffic, 149.3M outgoing IP traffic.
Jun 02 02:32:43 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 10min 56.853s CPU time over 8min 20.867s wall clock time, 27.5G memory peak, 1.2M read from disk, 994.4M incoming IP traffic, 168.9M outgoing IP traffic.
Jun 03 04:54:24 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 11min 46.965s CPU time over 11min 33.521s wall clock time, 27.5G memory peak, 3.5M read from disk, 1002.4M incoming IP traffic, 172.4M outgoing IP traffic.
Jun 04 09:05:16 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 12min 1.144s CPU time over 11min 29.549s wall clock time, 27.3G memory peak, 1.7M read from disk, 991.9M incoming IP traffic, 169.9M outgoing IP traffic.
Jun 05 11:19:02 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 12min 41.153s CPU time over 13min 59.676s wall clock time, 27.5G memory peak, 1.2M read from disk, 992.9M incoming IP traffic, 170.9M outgoing IP traffic.
Jun 06 07:49:50 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 11min 9.153s CPU time over 8min 31.081s wall clock time, 27.3G memory peak, 95.4M read from disk, 964M incoming IP traffic, 152.9M outgoing IP traffic.
Jun 07 10:08:57 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 11min 14.603s CPU time over 8min 35.812s wall clock time, 27.3G memory peak, 340K read from disk, 935.3M incoming IP traffic, 144.4M outgoing IP traffic.
Jun 08 10:01:02 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 11min 788ms CPU time over 8min 12.432s wall clock time, 27.4G memory peak, 3.6M read from disk, 946.3M incoming IP traffic, 152.7M outgoing IP traffic.
Jun 09 07:34:54 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 12min 19.105s CPU time over 12min 869ms wall clock time, 27.5G memory peak, 175M read from disk, 1010.2M incoming IP traffic, 172.5M outgoing IP traffic.
Jun 10 06:51:12 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 11min 13.910s CPU time over 8min 30.762s wall clock time, 27.8G memory peak, 15.7M read from disk, 949.5M incoming IP traffic, 152.3M outgoing IP traffic.
Jun 11 12:15:26 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 11min 45.493s CPU time over 11min 2.841s wall clock time, 27.6G memory peak, 128K read from disk, 957.6M incoming IP traffic, 157.4M outgoing IP traffic.
Jun 12 15:05:21 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 12min 31.497s CPU time over 13min 48.742s wall clock time, 27.5G memory peak, 965M incoming IP traffic, 162.8M outgoing IP traffic.
Jun 13 17:57:18 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 10min 57.415s CPU time over 8min 15.967s wall clock time, 28G memory peak, 3.2M read from disk, 918.2M incoming IP traffic, 138.5M outgoing IP traffic.
Jun 14 22:05:52 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 11min 12.270s CPU time over 8min 55.164s wall clock time, 27.6G memory peak, 2.1M read from disk, 964.1M incoming IP traffic, 159.9M outgoing IP traffic.
Jun 15 06:12:58 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 10min 56.567s CPU time over 8min 21.027s wall clock time, 27.6G memory peak, 15.4M read from disk, 959M incoming IP traffic, 151.4M outgoing IP traffic.
Jun 16 05:38:25 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 10min 42.588s CPU time over 11min 53.341s wall clock time, 27.7G memory peak, 5.1M read from disk, 988M incoming IP traffic, 171M outgoing IP traffic.
Jun 18 19:48:04 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 24min 54.429s CPU time over 22min 11.312s wall clock time, 27.4G memory peak, 512K read from disk, 959.1M incoming IP traffic, 149.7M outgoing IP traffic.
Jun 19 04:40:14 pluto systemd[1]: update-nixos-26.05-small.service: Consumed 37min 54.959s CPU time over 34min 30.494s wall clock time, 27.6G memory peak, 584K read from disk, 1G incoming IP traffic, 159.1M outgoing IP traffic.
14:15:06
@hexa:lossy.networkhexathe current run is at over 50 mins14:15:28
@hexa:lossy.networkhexa it takes a long time in nix-chanel-index 14:15:53
@hexa:lossy.networkhexaRedacted or Malformed Event14:15:58
@hexa:lossy.networkhexa K900 https://seashells.io/v/M6tM5udz 14:17:38
@hexa:lossy.networkhexa
● update-nixos-26.05-small.service - Update Channel nixos-26.05-small
     Loaded: loaded (/etc/systemd/system/update-nixos-26.05-small.service; linked; preset: ignored)
     Active: activating (start) since Sat 2026-06-20 13:21:09 UTC; 59min ago
14:20:45
@joerg:thalheim.ioMic92 hexa (signing key rotation when): Is this running nix-index? 14:28:05
@dramforever:matrix.orgdramforeverquestion, is it true that all the new bad paths are uploaded with zstd compression?14:42:04
@hexa:lossy.networkhexayeah14:42:21
@dramforever:matrix.orgdramforeveri seem to remember someone mentioning that we switched to the new runner alongside switching to zstd but i can't find it immediately14:42:24
@dramforever:matrix.orgdramforeverah thanks14:42:27
@hexa:lossy.networkhexa
      remoteStoreAddr = [
        "s3://nix-cache?${
          lib.concatStringsSep "&" [
            "secret-key=/var/lib/hydra/queue-runner/keys/cache.nixos.org-1/secret"
            "write-nar-listing=1"
            "compression=zstd"
            "compression-level=19"
            "ls-compression=zstd"
            "log-compression=zstd"
            "index-debug-info=true"
          ]
        }"
      ];
14:43:14
@dramforever:matrix.orgdramforeverokay, going through the keys list, it looks like we haven't grabbed many zstd compressed paths anyway, only ~100, which we'll just delete and redo next time and it should be all good14:46:28
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)

Some cursed build failure: https://hydra.nixos.org/build/331701865/nixlog/1

Doesn’t seem like a nix build issue, since other platforms are fine. Will try to take a look later

18:26:11
@hexa:lossy.networkhexayeah, that's some early gc18:27:20
@hexa:lossy.networkhexawas fixed in the last deploy18:27:24
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)Thanks. Will kick it then18:27:59
@hexa:lossy.networkhexahttps://github.com/NixOS/hydra/commit/09ee2bd5d39f49b8238e7b204d5e2e1c56808bc318:28:22
@dramforever:matrix.orgdramforevergood ol' gc race18:36:41
@dramforever:matrix.orgdramforeveroh already fixed? nice18:37:23

Show newer messages


Back to Room ListRoom Version: 6