!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

482 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.151 Servers

Load older messages


SenderMessageTime
18 Jun 2026
@joerg:thalheim.ioMic92thx08:03:10
@joerg:thalheim.ioMic92https://github.com/NixOS/hydra/issues/1805#issuecomment-4739539396 This might explain a few of the aborts08:10:05
@vmfunc:mozilla.orgvmfunc joined the room.08:17:29
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)

https://matrix.to/#/!CTCrFzsBPYmDLmrja4%3A0upti.me/%24olty6gvo6UpfhC5zyAr0LPIlO3U6CffpDZGhIAS89Ho

Nixpkgs isn’t faring too well too

08:18:56
@vmfunc:mozilla.orgvmfunchiya, ci looks broken repo-wide right now on nixpkgs. the ci/pinned.json bootstrap can't realise its tooling, hash mismatch on nghttp3-1.15.0 from cache.nixos.org (specified 157s6q…, got 07svi8…), no substituter, which takes out lint / parse, lint / nixpkgs-vet and eval / x86_64-darwin. it's been hitting PRs at random depending on how warm the runner's cache is. seeing it on #532911, #532912, and my #532910. known? pretty sure this is linked to cache08:18:59
@vmfunc:mozilla.orgvmfunc changed their profile picture.08:19:39
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)
In reply to @vmfunc:mozilla.org
hiya, ci looks broken repo-wide right now on nixpkgs. the ci/pinned.json bootstrap can't realise its tooling, hash mismatch on nghttp3-1.15.0 from cache.nixos.org (specified 157s6q…, got 07svi8…), no substituter, which takes out lint / parse, lint / nixpkgs-vet and eval / x86_64-darwin. it's been hitting PRs at random depending on how warm the runner's cache is. seeing it on #532911, #532912, and my #532910. known? pretty sure this is linked to cache
https://github.com/NixOS/hydra/issues/1804
08:19:39
@vmfunc:mozilla.orgvmfunci see08:20:27
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)Whats really scary is that the nixpkgs didn’t get bumped in CI?08:23:12
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)Last bump was https://github.com/NixOS/nixpkgs/pull/530903?08:24:17
@k900:0upti.meK900I wonder if it's uploading everything it builds08:25:01
@k900:0upti.meK900 Not just everything it doesn't have 08:25:05
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)Did existing NARs also get sprayed with garbage?08:25:07
@k900:0upti.meK900 And then something something race something 08:25:11
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)Lemme check the last-modified on the narinfo08:25:17
@vmfunc:mozilla.orgvmfunc changed their profile picture.08:25:19
@vmfunc:mozilla.orgvmfunc investigating rn, drafting a comment give me a moment 08:26:34
@r-burns:matrix.orgRyan Burnsthe texliveFull failure that raised that issue is a package that hasn't changed since 2026-06-10, so I'd say yes - perhaps a new invalid narinfo was pushed for it08:27:28
@vmfunc:mozilla.orgvmfunccache seems repaired now08:28:03
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)

On the /nix/store/i8ykl6jr2x9x3bqv2m8l6gn6i2246lyr-nghttp3-1.15.0 side something fishy is definitely going on:

curl https://cache.nixos.org/nar/157s6qjhidk6asyg2qg6ga6s5a78hwg2al9dyaw8sxx8759q2qx6.nar.zst -v |& grep "last-modified:"
last-modified: Thu, 18 Jun 2026 06:34:05 GMT
curl https://cache.nixos.org/i8ykl6jr2x9x3bqv2m8l6gn6i2246lyr.narinfo -v |& grep "last-modified:"
last-modified: Wed, 17 Jun 2026 06:18:34 GMT
08:31:44
@marie:marie.cologneMarie joined the room.08:37:05
@vcunat:matrix.orgVladimír Čunát All these .narinfo do have Compression: zstd 08:37:46
@vcunat:matrix.orgVladimír Čunátso that means that the .narinfo were also uploaded very recently, right?08:38:01
@vmfunc:mozilla.orgvmfuncmy edge serves the nar.zst as last-modified wed 17 jun 06:18, sergei's serves thu 18 jun 06:3408:38:03
@k900:0upti.meK900 So we're definitely double uploading then 08:38:19
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)Yeah, but that can't be right08:38:27
@vcunat:matrix.orgVladimír ČunátThe FOD from texlive was old, surely.08:38:40
@vmfunc:mozilla.orgvmfuncso it's not local cache warmth like i first guessed, it's edge inconsistency08:38:46
@joerg:thalheim.ioMic92If you have theories, please put them in the issue.08:38:49
@joerg:thalheim.ioMic92I will not be able to read everything in this chat08:39:00

Show newer messages


Back to Room ListRoom Version: 6