| 6 Oct 2021 |
 Vladimír Čunát | I assume that you correctly see cache.nixos.org. CNAMEd to dualstack.v2.shared.global.fastly.net. and it's some local problem with Fastly. (On my end I'm getting different IPs and cert seems accepted.) | 21:32:11 |
| Vladimír Čunát | $ openssl s_client -servername cache.nixos.org -connect 195.201.36.118:443
[...]
Certificate chain
0 s:CN = nextcloud.isardvdi.com
i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
i:O = Digital Signature Trust Co., CN = DST Root CA X3
| 21:34:01 |
| Vladimír Čunát | That's not just expired but very wrong cert. | 21:34:11 |
 vidister / fiona | Yeah, I don't know why it resolved to that IP, I can't get it to resolve that again, but it's still in the cache on my computer. dafuq_ | 21:36:21 |
| vidister / fiona | * Yeah, I don't know why it resolved to that IP, I can't get it to resolve that again, but it's still in the cache on my computer. dafuq? | 21:36:25 |
| Vladimír Čunát | The IP belongs to Hetzner, according to whois. | 21:37:13 |
 Sandro | Restart systemd-resolved maybe? | 21:37:20 |
| Vladimír Čunát | It really sounds like some DNS problem near your end. | 21:38:13 |
| Sandro | In reply to @vcunat:matrix.org
The IP belongs to Hetzner, according to whois. The sources of this data are not always up to date and in the past I already got really bogus results. Like 3 results from 3 services about location, owner and so on | 21:38:34 |
| Vladimír Čunát | ... though theoretically it is possible to be some Fastly DNS problem. | 21:38:40 |
| vidister / fiona | I'm not running systemd-resolved and the machine is using 1.1.1.1/8.8.8.8. I don't know where that result came from | 21:39:11 |
| vidister / fiona | this is so weird | 21:39:14 |
| vidister / fiona | well, thanks for helping, I'm super confused and I'll just flush my dns cache and stop investigating here.. | 21:40:44 |
| Vladimír Čunát | Perhaps, if it seems like a one-time issue, but it's weird. | 21:42:27 |
| 7 Oct 2021 |
 Jonas Chevalier | there is always the cosmic ray that can flip a bit :) | 08:12:02 |
| Jonas Chevalier | I'm investigating channel updates. nixos-unstable stopped pushing. | 08:12:52 |
| Jonas Chevalier | * I'm investigating channel updates. nixos-unstable stopped pushing. (investigation: https://github.com/NixOS/nixos-org-configurations/issues/180) | 08:17:03 |
| Jonas Chevalier | did anybody cancel builds recently? It looks like it was just caused by a bunch of cancelled builds. | 08:44:20 |
| Vladimír Čunát | This case was probably me. | 08:53:57 |
| Vladimír Čunát | If some jobs are required for channel updates to succeed, shouldn't we make the critical job (tested here) depend on all those jobs? | 08:54:39 |
| Vladimír Čunát | * This case was probably me. | 08:55:23 |
| Vladimír Čunát | Well, I wonder where to reply... seeing the question on one issue and two chat channels. | 08:56:36 |
| Vladimír Čunát | ☝️ Jonas Chevalier | 08:57:23 |
| Jonas Chevalier | that's fine, I'm just trying to better understand what happened | 08:58:38 |
| Jonas Chevalier | I'm not sure super familiar with Hydra, and the channel-update logic | 08:59:37 |
| Jonas Chevalier | Vladimír Čunát: what are typical conditions where builds get canceled? | 09:00:31 |
| Vladimír Čunát | I do it in a few cases | 09:01:28 |
| Vladimír Čunát |
- replaced builds: if we have newer builds of the same jobs (scheduled) and keeping the older ones doesn't seem very useful anymore. (makes sense when they're many, especially if combined with high load on Hydra)
| 09:02:52 |
| Vladimír Čunát |
- channel waiting on bad test. Sometimes the channel would update, but it waits to finish all builds. And sometimes there's a few tests that never succeeded recently and just wait for a long time-out. Cancelling those stragglers can speed up channel update. (makes sense if the channel is quite old currently, e.g. after some period of channel blockers)
| 09:04:34 |
| Jonas Chevalier | Is anybody else keeping a pulse on Hydra like you do? | 09:11:43 |
