13 Dec 2022 |
m1cr0man | 103 attempts later and not a single one has failed 😕 | 22:38:37 |
m1cr0man | Okay so being a bit smarter with this debugging, I am looking through the build logs on Hydra for successful builds and checking where any retry logic was triggered and how many times. if you search this build https://hydra.nixos.org/build/201652934/nixlog/1 for "s_client -brief" You will see an instance in the first few matches in which it has to be retried 3 consecutive times (and works on the third). The method performing this is here: https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/acme.nix#L407-L418 and is configured for 3 retries. You can also see the webserver giving the error client closed connection while waiting for request on 2 of the 3 attempts.
I think I need to increase the delays + number of retries for this method and any others that are waiting on web responses. Even from that log, I can't see any reason the server wasn't able to respond or why the client had sent a partial request. Hopefully this will be sufficient to stop the failures.
| 22:59:25 |
m1cr0man | Welp, here's a PR https://github.com/NixOS/nixpkgs/pull/205983 hopefully this does the trick. I have good evidence to support increased retries + delays solving the issue. | 23:44:37 |
14 Dec 2022 |
m1cr0man | Awh dammit I just realized that vscode auto formatted it 🤦🤦 will fix tomorrow | 00:22:06 |
| Alesya Huzik joined the room. | 11:13:02 |
21 Dec 2022 |
| @thatsnomoon_343:matrix.org joined the room. | 05:01:47 |
| @thatsnomoon_343:matrix.org left the room. | 20:16:41 |
22 Dec 2022 |
m1cr0man | Thanks for the merge Raito! 🙂 | 03:25:12 |
raitobezarius | with pleasure | 04:15:18 |
raitobezarius | thanks for all the amazing work on ACME ;) | 04:15:28 |
24 Dec 2022 |
K900 | https://hydra.nixos.org/build/202864292/nixlog/5 | 07:51:42 |
K900 | It broke again :( | 07:51:47 |
raitobezarius | In reply to @k900:0upti.me https://hydra.nixos.org/build/202864292/nixlog/5 did it? | 20:29:55 |
hexa | Probably fine after restart | 21:11:52 |
25 Dec 2022 |
| @ahsmha:matrix.org left the room. | 10:34:49 |
26 Dec 2022 |
m1cr0man | Huh I think my Matrix client was broke, those messages from Saturday just appeared. | 14:17:22 |
31 Dec 2022 |
hexa | 😲 | 19:49:12 |
1 Jan 2023 |
| @v0id:nltrix.net left the room. | 12:05:26 |
11 Jan 2023 |
hexa | so I had to restore a host yesterday and I could've restored the certificates from a backup, but didn't | 15:55:52 |
hexa | the acme module recreated all certs no problemo | 15:56:04 |
hexa | on the first try | 15:56:08 |
hexa | mind you, the host has 10 different ones 🙂 | 15:56:25 |
hexa | 🙏 | 15:56:38 |