!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

468 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.147 Servers

Load older messages


SenderMessageTime
27 Jun 2026
@hexa:lossy.networkhexa (signing key rotation when)🤡14:22:50
@emilazy:matrix.orgemilydo we really need to GC that often?14:23:24
@hexa:lossy.networkhexa (signing key rotation when)we can try and reduce14:23:56
@k900:0upti.meK900I assume those have less disk also?14:23:57
@emilazy:matrix.orgemilyoh I just mean in general14:24:03
@hexa:lossy.networkhexa (signing key rotation when)but fixing gc was annoying for the small macs with little disk14:24:08
@hexa:lossy.networkhexa (signing key rotation when)the linux builders have between 2x2 and 2x4TB NVME14:24:46
@hexa:lossy.networkhexa (signing key rotation when)the macs have like 256 for the hetzner ones14:24:58
@hexa:lossy.networkhexa (signing key rotation when)and more for the oakhost ones14:25:04
@hexa:lossy.networkhexa (signing key rotation when)we can try fast-nix-gc14:26:19
@hexa:lossy.networkhexa (signing key rotation when)what would it fix? I guess gcing faster would be benefitial to the nodes anyhow14:26:59
@hexa:lossy.networkhexa (signing key rotation when)but the window during which we'd gc would become smaller14:27:10
@emilazy:matrix.orgemilyyeah I'm just surprised that even Linux builders fill up that quickly14:27:17
@emilazy:matrix.orgemilydoing it twice an hour would be annoying if you wanted to drain the builders presumably14:27:38
@hexa:lossy.networkhexa (signing key rotation when)I don't think we tuned them as much14:27:41
@hexa:lossy.networkhexa (signing key rotation when)we like to keep them fuller so they have deps they might need later14:27:59
@emilazy:matrix.orgemilywonder how easy it is to just make the builds take the GC lock14:28:01
@hexa:lossy.networkhexa (signing key rotation when)and with fast-nix-gc we can tell it to keep newer things alive14:28:17
@emilazy:matrix.orgemily Sergei Zimmerman (xokdvium): how easy would it be to have an option for "refuse to try building something if some of its outputs are present" that defaults on if fallback paths would be used (i.e. macOS, unsandboxed Linux) and then we could set it on for all of Hydra? 14:31:17
@emilazy:matrix.orgemilyI guess that still doesn't handle the case where some of the outputs reached the cache but others didn't, and another machine tries doing the build to satisfy another output, but hasn't substituted the outputs that are in the cache14:31:40
@emilazy:matrix.orgemilythen the cache still gets frankenbuilds14:31:46
@joerg:thalheim.ioMic92
In reply to @hexa:lossy.network
we can try fast-nix-gc
It also can skip recently added paths. And we can actually block on gc.lock now
14:33:49
@hexa:lossy.networkhexa (signing key rotation when)that's what I meant here14:34:09
@hexa:lossy.networkhexa (signing key rotation when)can you pick your connection pooling change on top of hydra.nixos.org?14:34:28
@hexa:lossy.networkhexa (signing key rotation when)so we can merge14:34:31
@joerg:thalheim.ioMic92* It also can skip recently added paths. And we can actually block on gc.lock now to not schedule new builds. It will now create more io14:34:35
@joerg:thalheim.ioMic92* It also can skip recently added paths. And we can actually block on gc.lock now to not schedule new builds. It will now create more io anyway, so might be for the best to not schedule new builds during that 14:35:44
@joerg:thalheim.ioMic92
In reply to @hexa:lossy.network
can you pick your connection pooling change on top of hydra.nixos.org?
I have on my laptop but not yet pushed. Need to wait for my train
14:36:22
@hexa:lossy.networkhexa (signing key rotation when)i fairness, we now also clean out logs much more reliably on darwin14:41:54
@hexa:lossy.networkhexa (signing key rotation when) everything in macs/common/workarounds.nix 14:42:15

Show newer messages


Back to Room ListRoom Version: 6