!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

374 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.114 Servers

Load older messages


SenderMessageTime
22 Jul 2025
@hexa:lossy.networkhexa (signing key rotation when)the traceroute not completing could be a red flag13:01:14
@drupol:matrix.org@drupol:matrix.orgOk... 13:01:14
@drupol:matrix.org@drupol:matrix.orgyeah that's so weird :(13:01:35
@drupol:matrix.org@drupol:matrix.orgI have that thing running for now 30 minutes:13:02:01
@drupol:matrix.org@drupol:matrix.orgimage.png
Download image.png
13:02:03
@drupol:matrix.org@drupol:matrix.orgKinda frustrating.13:02:10
@drupol:matrix.org@drupol:matrix.orgThey do everything to keep their workers at home :D13:02:18
@hexa:lossy.networkhexa (signing key rotation when)also a very funny route13:03:05
@hexa:lossy.networkhexa (signing key rotation when)given that you are presumably sitting in brussels, but handing of the traffic to british telecom13:03:16
@hexa:lossy.networkhexa (signing key rotation when)for a detour via londoin13:03:20
@hexa:lossy.networkhexa (signing key rotation when)* for a detour via london13:03:22
@hexa:lossy.networkhexa (signing key rotation when)only to go back to amsterdam13:03:34
@drupol:matrix.org@drupol:matrix.orgyes indeed, sitting in brussels at the moment13:03:39
@hexa:lossy.networkhexa (signing key rotation when)* given that you are presumably sitting in brussels, but handing off the traffic to british telecom13:03:53
@drupol:matrix.org@drupol:matrix.orgIs there anything else I could do to try to spot where the issue comes from?13:04:00
@hexa:lossy.networkhexa (signing key rotation when)this really looks like the reverse path from fastly to your AS is broken13:04:16
@hexa:lossy.networkhexa (signing key rotation when)the solution is … peer with fastly somewhere and cut out the middleman13:04:42
@drupol:matrix.org@drupol:matrix.orgWhat worries me is this: the behaviour is strange: downloading just 30 MB takes around 15 minutes. It seems to follow a repeating pattern: the connection works normally for 1–2 seconds, then pauses for about 30 seconds, and this cycle continues indefinitely until the end. 13:04:52
@hexa:lossy.networkhexa (signing key rotation when)yeah, a route can flap13:05:15
@drupol:matrix.org@drupol:matrix.org

peer with fastly somewhere and cut out the middleman

how can I do that?

13:05:43
@hexa:lossy.networkhexa (signing key rotation when)https://bgp.tools/prefix/147.67.128.0/17#connectivity13:05:48
@hexa:lossy.networkhexa (signing key rotation when)single homed behind vodafone 🫠13:05:56
@hexa:lossy.networkhexa (signing key rotation when)if you're not at any exchange you likely cannot13:06:48
@drupol:matrix.org@drupol:matrix.orgI guess the network here is quite protected indeed.13:07:15
@drupol:matrix.org@drupol:matrix.organyway, thanks for the help and the link I'll keep it :)13:09:12
@federicodschonborn:matrix.org@federicodschonborn:matrix.org changed their display name from The Worm 🏳️‍🌈 (he/they) to Wormy McWormface 🏳️‍🌈 (he/they).18:27:19
@solidhal:matrix.orgsolideva joined the room.20:41:27
23 Jul 2025
@vcunat:matrix.orgvcunat

In the past couple hours we get lots of

mimas hydra-queue-runner[1040202]: AWS error '' (curlCode: 28, Timeout was reached; request id: ), will retry in NN ms

I see nothing at https://health.aws.amazon.com/health/status

10:28:29
@vcunat:matrix.orgvcunatHmm, we only fit less than two days of log for that unit, so who knows how rare this is over long term, but the build throughput does seem unusually low in the past couple hours (which is why I was looking at logs).10:31:15
@vcunat:matrix.orgvcunat *

In the past couple hours we get thousands of

mimas hydra-queue-runner[1040202]: AWS error '' (curlCode: 28, Timeout was reached; request id: ), will retry in NN ms

I see nothing at https://health.aws.amazon.com/health/status

10:32:16

Show newer messages


Back to Room ListRoom Version: 6