!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

469 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.146 Servers

Load older messages


SenderMessageTime
17 Jun 2026
@k900:0upti.meK900Because the store path already exists14:15:00
@hexa:lossy.networkhexa (signing key rotation when)https://github.com/NixOS/hydra/blob/f69a55cf3c73146ba4ac4d5e7d39377a226d85fb/subprojects/hydra-builder/src/state.rs#L629-L639 upload_nars called here14:19:26
@hexa:lossy.networkhexa (signing key rotation when)and I don't see any retry logic14:20:09
@hexa:lossy.networkhexa (signing key rotation when)not in upload_nars, not in upload_nars_presigned14:20:39
@hexa:lossy.networkhexa (signing key rotation when)we deployed another update to fix grpc connection aborts from nginx16:46:27
@k900:0upti.meK900 https://hydra.nixos.org/build/331658388 18:03:29
@k900:0upti.meK900Another unsupported system situation18:03:33
@hexa:lossy.networkhexa (signing key rotation when)
Download
23:22:23
@hexa:lossy.networkhexa (signing key rotation when)uhhhhhhh23:22:27
@hexa:lossy.networkhexa (signing key rotation when)160k rebuilds seems a bit excessive here23:22:58
@hexa:lossy.networkhexa (signing key rotation when)
❯ nix-diff /nix/store/m74651b793zgyyvlk9gx7v1cl1ywslib-hello-2.12.3.drv  /nix/store/gh9rl6g6jw0kvkzi3mh40ig2phffrwjz-hello-2.12.3.drv
- /nix/store/m74651b793zgyyvlk9gx7v1cl1ywslib-hello-2.12.3.drv:{out}
+ /nix/store/gh9rl6g6jw0kvkzi3mh40ig2phffrwjz-hello-2.12.3.drv:{out}
• The input derivation named `hello-2.12.3.tar.gz` differs
  - /nix/store/jk8228a88vjsa0zk004sgdhxp5g4pnnv-hello-2.12.3.tar.gz.drv:{out}
  + /nix/store/b1nwvxqz0prj6kqvwfs7i8d7sk40m4w7-hello-2.12.3.tar.gz.drv:{out}
  • The input derivation named `mirrors-list` differs
    - /nix/store/84w0zzmqzl2bn5s4rx7axsh9znws1rdf-mirrors-list.drv:{out}
    + /nix/store/qkrfwhcrn7af1w11gxh3m45xjsgxlrxz-mirrors-list.drv:{out}
    • The environments do not match:
        + texhistoric=https://ftp.math.utah.edu/pub/tex/historic/ https://texlive.info/historic/ https://ftp.tu-chemnitz.de/pub/tug/historic/ https://pi.kwarc.info/historic/ https://mirrors.tuna.tsinghua.edu.cn/tex-historic-archive/ https://mirror.nju.edu.cn/tex-historic/ ftp://tug.org/texlive/historic/
  • Skipping environment comparison
• Skipping environment comparison
23:26:37
@hexa:lossy.networkhexa (signing key rotation when)mirror list change looks sus23:26:41
@hexa:lossy.networkhexa (signing key rotation when)https://github.com/NixOS/nixpkgs/pull/51848023:27:53
@hexa:lossy.networkhexa (signing key rotation when)yup, 77456b834919be3e6e8ed00bd26eebd32046dc8e causes a rebuild for hello23:31:20
@hexa:lossy.networkhexa (signing key rotation when)heh23:32:39
@hexa:lossy.networkhexa (signing key rotation when)
Download
23:32:47
18 Jun 2026
@hexa:lossy.networkhexa (signing key rotation when)ok, trying out iocaine instead of anubis on hydra00:30:19
@hexa:lossy.networkhexa (signing key rotation when)if you see random markov shit instead of hydra, feel free to reach out00:30:32
@hexa:lossy.networkhexa (signing key rotation when)but the real benefit is that the anubis page where you generally wait goes away with this00:30:57
@vcunat:matrix.orgvcunatThe situation doesn't seem sustainable to me for more than a couple days. Even -small channels get blocked because of the abortions and retrying doesn't seem to help, e.g.: https://hydra.nixos.org/build/33165835804:39:50
@vcunat:matrix.orgvcunat * The situation doesn't seem sustainable to me for more than a couple days. Even nixos-26.05-small channel gets blocked because of the abortions and retrying doesn't seem to help, e.g.:
https://hydra.nixos.org/build/331658358
04:40:28
@vcunat:matrix.orgvcunatI expect it was this again.05:24:39
@vcunat:matrix.orgvcunat* Rebuilds: I expect it was this again.05:24:50
@r-burns:matrix.orgr-burnsI think cache.nixos.org is somehow serving the wrong path for a package, resulting in NAR mismatches. I assume this is the proper place to report this so someone with infra access can nuke the invalid cache entry? (If this is indeed what's happening - not super sure because I've never seen this before!) https://discourse.nixos.org/t/error-when-rebuilding-with-texlivefull-package/7835607:14:08
@k900:0upti.meK900 OK now that 07:15:37
@k900:0upti.meK900Is a fucking problem07:15:39
@k900:0upti.meK900 @hexa (signing key rotation when) picnic 07:15:58
@vcunat:matrix.orgvcunatA bug in the queue-builder?07:23:21
@vcunat:matrix.orgvcunatHaving a completely different package in the cached NAR is something I haven't heard of yet.07:24:00
@vcunat:matrix.orgvcunat* Having a completely different package in the cached NAR is something I haven't heard of yet, so the timing is suspicious.07:24:23

Show newer messages


Back to Room ListRoom Version: 6