!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

477 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.149 Servers

Load older messages


SenderMessageTime
10 Jun 2026
@k900:0upti.meK900Cancelled nixos-unstable{,-small} from 5 minutes ago06:30:08
@k900:0upti.meK900 For https://github.com/NixOS/nixpkgs/pull/530238 06:30:13
@k900:0upti.meK900Will restart once that lands06:30:28
@k900:0upti.meK900Ughhh06:54:58
@k900:0upti.meK900 @vcunat can you cancel the evals 06:55:01
@vcunat:matrix.orgVladimír ČunátI already did.06:56:42
@k900:0upti.meK900Thanks06:58:14
@vcunat:matrix.orgVladimír ČunátI haven't confirmed any failure in the Hydra's UI since then. I looked specifically at ISO builds, as those seemed to suffer abnormally from this. I don't know if this evidence has statistical significance already, though.14:34:52
@vcunat:matrix.orgVladimír ČunátLooking at hydra-queue-runner logs, we have lots of cases even today where the attempt counter goes over 10/32.14:37:32
@vcunat:matrix.orgVladimír ČunátWait... "retrying in 2346603 ms". Thats 39 minutes. 14:39:32
@vcunat:matrix.orgVladimír Čunát* Wait... "retrying in 2346603 ms". That's 39 minutes. 14:39:42
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)Mhm yeah I mentioned that we should cap the retry time limit, I think that on master but not 2.3414:40:30
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)Or even just make these failures exempt, those don’t need exponential back off at all14:41:06
@vcunat:matrix.orgVladimír ČunátIf the cap is 32 attempts, the backoff goes up to 2^32? 🤣14:42:21
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)😵‍💫14:42:33
@hexa:lossy.networkhexawelp14:57:14
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)Lemme add a no delay handling for these sorts of errors15:02:13
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)https://github.com/aws/aws-cli/issues/401#issuecomment-88380435915:02:22
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)On 2.34 it would overflow apparently15:16:24
@xokdvium:matrix.orgSergei Zimmerman (xokdvium)Well, if it had the chance to that is15:16:32
@xokdvium:matrix.orgSergei Zimmerman (xokdvium) vcunat: I've pushed this https://github.com/NixOS/nix/pull/15949/changes/15ee316650a282432676207e472de8d2f7a77f6c.
I hate everything about this, but such is AWS apparently?
15:19:43
@hexa:lossy.networkhexaloving it15:20:38
11 Jun 2026
@hexa:lossy.networkhexahttps://meet.cccda.de/nix-osin-fra16:00:54
@jfly:matrix.orgJeremy Fleischman (jfly) Sorry to miss, travelling this week. 16:34:36
13 Jun 2026
@debtquity:matrix.orgdebtquity set a profile picture.03:23:02
@klea:catgirl.cloudklea (she/her) joined the room.21:45:50
14 Jun 2026
@orzklv:floss.uz@orzklv:floss.uz left the room.23:32:38
15 Jun 2026
@k900:0upti.meK900Uhh is hydra dead07:22:01
@k900:0upti.meK900 Yeah I think hydra's dead 07:23:01
@k900:0upti.meK900OK wait never mind I think I got myself IP banned somehow07:23:25

Show newer messages


Back to Room ListRoom Version: 6