!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

271 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.86 Servers

Load older messages


SenderMessageTime
19 Oct 2024
@hexa:lossy.networkhexa (signing key rotation when)booted the arm builders there as well22:33:41
@hexa:lossy.networkhexa (signing key rotation when)we should be good then22:33:45
@vcunat:matrix.orgvcunatx86 ones don't seem good yet. Maybe they took old version because the whole eval wasn't finished?22:36:47
@vcunat:matrix.orgvcunat * x86 ones don't seem good yet (no stuff being built on them). Maybe they took old version because the whole eval wasn't finished?22:37:14
@hexa:lossy.networkhexa (signing key rotation when)I checked one before booting the others22:37:39
@hexa:lossy.networkhexa (signing key rotation when)the eval doesn't need to be finished apparently22:37:45
@hexa:lossy.networkhexa (signing key rotation when)
❯ ssh root@80ad8720.packethost.net
[root@nixos:~]# nix --version
nix (Nix) 2.24.9
22:38:36
@hexa:lossy.networkhexa (signing key rotation when)random sample22:38:45
@hexa:lossy.networkhexa (signing key rotation when)hm ok22:43:32
@hexa:lossy.networkhexa (signing key rotation when)Oct 19 22:43:21 rhea hydra-queue-runner[3665067]: possibly transient failure building ‘/nix/store/2jaiapw5i9lwd1ih1vxsrcks2jjwhbip-clang-at-least-16-LLVMgold-path.patch.drv’ on ‘root@5e4f5789.packethost.net’: error: invalid character in Base64 string: '' 22:43:35
@hexa:lossy.networkhexa (signing key rotation when)Oct 19 22:43:21 rhea hydra-queue-runner[3665067]: possibly transient failure building ‘/nix/store/7nglzcyk898mr827mcvnyka9fs7q1j45-directx-headers-armv6l-unknown-linux-gnueabihf-1.614.0.drv’ on ‘root@5e4f5789.packethost.net’: error: invalid character in Base64 string: '' 22:43:38
@hexa:lossy.networkhexa (signing key rotation when)Oct 19 22:43:21 rhea hydra-queue-runner[3665067]: possibly transient failure building ‘/nix/store/mc1rrbgi3s9ibhwn834prl3bklf5kcv1-rust-cbindgen-0.27.0.drv’ on ‘root@5e4f5789.packethost.net’: error: invalid character in Base64 string: '' 22:43:42
@hexa:lossy.networkhexa (signing key rotation when)Oct 19 22:43:21 rhea hydra-queue-runner[3665067]: possibly transient failure building ‘/nix/store/9p6rqdbcy68a8nn3s1alygyq473wfj9m-libXxf86vm-armv6l-unknown-linux-gnueabihf-1.1.5.drv’ on ‘root@5e4f5789.packethost.net’: error: invalid character in Base64 string: '' 22:43:44
@hexa:lossy.networkhexa (signing key rotation when)
❯ ssh root@5e4f5789.packethost.net -i ~/.ssh/id_nix-infra

[root@nixos:~]# nix --version
nix (Nix) 2.24.9
22:44:22
@hexa:lossy.networkhexa (signing key rotation when)that's great22:44:32
@hexa:lossy.networkhexa (signing key rotation when)so we're seeing a hydra compat issue maybe?22:44:47
@hexa:lossy.networkhexa (signing key rotation when) John Ericson: ptal when you have time 22:51:16
@hexa:lossy.networkhexa (signing key rotation when)
Derivation
  { outputs =
      fromList
        [ ( "out"
          , DerivationOutput
              { path =
                  "/nix/store/gfwnjwyg9izy36kxfqkkvg60n845k6wq-rust-cbindgen-0.27.0"
              , hashAlgo = ""
              , hash = ""
              }
          )
        ]
  , inputDrvs =
      fromList
        [ ( "/nix/store/2zbdh7ww5jpmy5kk2s0xlc3nhsmlmjm0-cargo-build-hook.sh.drv"
          , fromList [ "out" ]
          )
        , ( "/nix/store/d4xqlxgkq99jfql5l8sybrh3jnn2zvj6-cargo-install-hook.sh.drv"
          , fromList [ "out" ]
          )
        , ( "/nix/store/diappn3n1z5acaqnk6kw0mj53pzwbw47-stdenv-linux.drv"
          , fromList [ "out" ]
          )
        , ( "/nix/store/ghxjjdi9g5dr10096sl200rr13h5s89w-source.drv"
          , fromList [ "out" ]
          )
        , ( "/nix/store/hrb2k9y27lsy7x0w2cms1cd3vb0z16bz-rustc-wrapper-1.81.0.drv"
          , fromList [ "out" ]
          )
        , ( "/nix/store/ixsi66s2prpj5wr0prwybr00xcnp9qz8-rust-cbindgen-0.27.0-vendor.tar.gz.drv"
          , fromList [ "out" ]
          )
        , ( "/nix/store/jm0a2r0bm29xhwj0z41rc98nrhi04jda-python3.12-cython-3.0.11.drv"
          , fromList [ "out" ]
          )
        , ( "/nix/store/k2390c0gza3gsiwdmsf2papzjd7pjxm9-auditable-cargo-1.81.0.drv"
          , fromList [ "out" ]
          )
        , ( "/nix/store/lnamq8bbafa0yzaf6awphri6s5pszcvk-cargo-setup-hook.sh.drv"
          , fromList [ "out" ]
          )
        , ( "/nix/store/nkj6x4ajnxz1w80mhi7sspjpkvgp3bpk-bash-5.2p32.drv"
          , fromList [ "out" ]
          )
        , ( "/nix/store/s7wkvim8lplr5k4zf0k083ivhwn0im2l-cmake-3.29.6.drv"
          , fromList [ "out" ]
          )
        , ( "/nix/store/xw8fmrxxbnqq1v0xwmjnfmvkf0c4h3xb-cargo-check-hook.sh.drv"
          , fromList [ "out" ]
          )
        ]
  , inputSrcs =
      fromList
        [ "/nix/store/nk6b2ckznjic5wj8ddw0wgdrn4mbz3lg-patch-registry-deps"
        , "/nix/store/v6x3cs394jgqfbi0a42pam708flxaphh-default-builder.sh"
        ]
  , platform = "x86_64-linux"
  , builder =
      "/nix/store/717iy55ncqs0wmhdkwc5fg2vci5wbmq8-bash-5.2p32/bin/bash"
  , args =
      [ "-e"
      , "/nix/store/v6x3cs394jgqfbi0a42pam708flxaphh-default-builder.sh"
      ]
  , env =
      fromList
        [ ( "PKG_CONFIG_ALLOW_CROSS" , "0" )
        , ( "__structuredAttrs" , "" )
        , ( "buildInputs" , "" )
        , ( "builder"
          , "/nix/store/717iy55ncqs0wmhdkwc5fg2vci5wbmq8-bash-5.2p32/bin/bash"
          )
        , ( "cargoBuildFeatures" , "" )
        , ( "cargoBuildNoDefaultFeatures" , "" )
        , ( "cargoBuildType" , "release" )
        , ( "cargoCheckFeatures" , "" )
        , ( "cargoCheckNoDefaultFeatures" , "" )
        , ( "cargoCheckType" , "release" )
        , ( "cargoDeps"
          , "/nix/store/j453bkbq1jxv6q8pw3cz0y2c60rsibdj-rust-cbindgen-0.27.0-vendor.tar.gz"
          )
        , ( "cargoHash"
          , "sha256-l4FgwXdibek4BAnqjWd1rLxpEwuMNjYgvo6X3SS3fRo="
          )
        , ( "checkFlags"
          , "--skip test_expand --skip test_bitfield --skip lib_default_uses_debug_build --skip lib_explicit_debug_build --skip lib_explicit_release_build"
          )
        , ( "cmakeFlags" , "" )
        , ( "configureFlags" , "" )
        , ( "configurePhase"
          , "runHook preConfigure\nrunHook postConfigure\n"
          )
        , ( "depsBuildBuild" , "" )
        , ( "depsBuildBuildPropagated" , "" )
        , ( "depsBuildTarget" , "" )
        , ( "depsBuildTargetPropagated" , "" )
        , ( "depsHostHost" , "" )
        , ( "depsHostHostPropagated" , "" )
        , ( "depsTargetTarget" , "" )
        , ( "depsTargetTargetPropagated" , "" )
        , ( "doCheck" , "1" )
        , ( "doInstallCheck" , "" )
        , ( "mesonFlags" , "" )
        , ( "name" , "rust-cbindgen-0.27.0" )
        , ( "nativeBuildInputs"
          , "/nix/store/nkly9ryl33j1038lsv7gc9mp92wvwh2b-auditable-cargo-1.81.0 /nix/store/4xylfir30ayji9imxxbzj4vxgd2fxbv1-cargo-build-hook.sh /nix/store/6b7mhw8m9yiqafc0yh36lry45j70mjfh-cargo-check-hook.sh /nix/store/s57n3fcgl7ddwvjv2wy9c86nx5haxkmk-cargo-install-hook.sh /nix/store/xyk9lscawjfn4a0cz052xc8f0x52d0cr-cargo-setup-hook.sh /nix/store/csadsvzmnzvb952kjky9ziinky5q8abr-rustc-wrapper-1.81.0 /nix/store/24gbi6mcw4c6jgzai9aaxx9qi2rxdb09-cmake-3.29.6 /nix/store/4xig4sp870s5ghgjx1vyp3q4pf31ws69-python3.12-cython-3.0.11"
          )
        , ( "out"
          , "/nix/store/gfwnjwyg9izy36kxfqkkvg60n845k6wq-rust-cbindgen-0.27.0"
          )
        , ( "outputs" , "out" )
        , ( "patchRegistryDeps"
          , "/nix/store/nk6b2ckznjic5wj8ddw0wgdrn4mbz3lg-patch-registry-deps"
          )
        , ( "patches" , "" )
        , ( "pname" , "rust-cbindgen" )
        , ( "postUnpack"
          , "eval \"$cargoDepsHook\"\n\nexport RUST_LOG=\n"
          )
        , ( "propagatedBuildInputs" , "" )
        , ( "propagatedNativeBuildInputs" , "" )
        , ( "src" , "/nix/store/b4xbka7yazbxnhmjwhk5pgl390x06jda-source" )
        , ( "stdenv"
          , "/nix/store/ncv68hjnidcd2bm5abkhklrijhn0cgn6-stdenv-linux"
          )
        , ( "strictDeps" , "1" )
        , ( "system" , "x86_64-linux" )
        , ( "version" , "0.27.0" )
        ]
  }
22:52:47
@hexa:lossy.networkhexa (signing key rotation when)example derivation that fails22:52:52
@hexa:lossy.networkhexa (signing key rotation when)I'm going to get some sleep now22:53:52
@hexa:lossy.networkhexa (signing key rotation when)one last thing: the arm builder at hetzner works with nix 2.18.8, while the one at EM with nix 2.24.9 does not22:55:20
@vcunat:matrix.orgvcunatI also give up for today.22:59:38
@hexa:lossy.networkhexa (signing key rotation when)tomorrow we might try to rollback hydra to the working 2.19 state on 24.05 23:00:47
@hexa:lossy.networkhexa (signing key rotation when)maybe that'll make things work, who knows23:00:59
20 Oct 2024
@hexa:lossy.networkhexa (signing key rotation when)ok, apparently I'm still awake00:14:50
@hexa:lossy.networkhexa (signing key rotation when)and puck came up with a fix00:15:06
@hexa:lossy.networkhexa (signing key rotation when)the EM hosts were the only ones affected, because they specify an ssh host key in their machine definition00:15:29
@hexa:lossy.networkhexa (signing key rotation when)that gets decoded once in hydra, then passed to nix, which tries to decode it again … which fails00:15:49
@hexa:lossy.networkhexa (signing key rotation when)so we are back in production00:16:07
@hexa:lossy.networkhexa (signing key rotation when)the builders are currently running on 2.18.8, but I reverted back to unpinning the nix version again00:16:36

Show newer messages


Back to Room ListRoom Version: 6