!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

482 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.151 Servers

Load older messages


SenderMessageTime
5 Jun 2026
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)I have no clue how x-amz-website-redirect-location works09:02:32
@dramforever:matrix.orgdramforeveri think it gets converted into a redirect at some point during the fastlyification09:03:45
@vcunat:matrix.orgVladimír Čunát eager-heisenberg.mac.nixos.org looked stuck for roughly 24h, rebooting. 11:05:32
7 Jun 2026
@fgrsnau:matrix.orgStefan joined the room.13:44:34
9 Jun 2026
@lyndeno:matrix.orgLyndeno joined the room.13:03:02
10 Jun 2026
@tomberek:matrix.orgtomberekupdate or news about https://github.com/NixOS/nix/pull/15949 ?03:50:13
@k900:0upti.meK900Cancelled nixos-unstable{,-small} from 5 minutes ago06:30:08
@k900:0upti.meK900 For https://github.com/NixOS/nixpkgs/pull/530238 06:30:13
@k900:0upti.meK900Will restart once that lands06:30:28
@k900:0upti.meK900Ughhh06:54:58
@k900:0upti.meK900 @vcunat can you cancel the evals 06:55:01
@vcunat:matrix.orgVladimír ČunátI already did.06:56:42
@k900:0upti.meK900Thanks06:58:14
@vcunat:matrix.orgVladimír ČunátI haven't confirmed any failure in the Hydra's UI since then. I looked specifically at ISO builds, as those seemed to suffer abnormally from this. I don't know if this evidence has statistical significance already, though.14:34:52
@vcunat:matrix.orgVladimír ČunátLooking at hydra-queue-runner logs, we have lots of cases even today where the attempt counter goes over 10/32.14:37:32
@vcunat:matrix.orgVladimír ČunátWait... "retrying in 2346603 ms". Thats 39 minutes. 14:39:32
@vcunat:matrix.orgVladimír Čunát* Wait... "retrying in 2346603 ms". That's 39 minutes. 14:39:42
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)Mhm yeah I mentioned that we should cap the retry time limit, I think that on master but not 2.3414:40:30
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)Or even just make these failures exempt, those don’t need exponential back off at all14:41:06
@vcunat:matrix.orgVladimír ČunátIf the cap is 32 attempts, the backoff goes up to 2^32? 🤣14:42:21
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)😵‍💫14:42:33
@hexa:lossy.networkhexawelp14:57:14
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)Lemme add a no delay handling for these sorts of errors15:02:13
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)https://github.com/aws/aws-cli/issues/401#issuecomment-88380435915:02:22
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)On 2.34 it would overflow apparently15:16:24
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)Well, if it had the chance to that is15:16:32
@xokdvium:matrix.orgSergei Zimmerman (xokdvium) vcunat: I've pushed this https://github.com/NixOS/nix/pull/15949/changes/15ee316650a282432676207e472de8d2f7a77f6c.
I hate everything about this, but such is AWS apparently?
15:19:43
@hexa:lossy.networkhexaloving it15:20:38
11 Jun 2026
@hexa:lossy.networkhexahttps://meet.cccda.de/nix-osin-fra16:00:54
@jfly:matrix.orgJeremy Fleischman (jfly) Sorry to miss, travelling this week. 16:34:36

Show newer messages


Back to Room ListRoom Version: 6