Okay so being a bit smarter with this debugging, I am looking through the build logs on Hydra for successful builds and checking where any retry logic was triggered and how many times. if you search this build https://hydra.nixos.org/build/201652934/nixlog/1 for "s_client -brief" You will see an instance in the first few matches in which it has to be retried 3 consecutive times (and works on the third). The method performing this is here: https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/acme.nix#L407-L418 and is configured for 3 retries. You can also see the webserver giving the error client closed connection while waiting for request on 2 of the 3 attempts.

I think I need to increase the delays + number of retries for this method and any others that are waiting on web responses. Even from that log, I can't see any reason the server wasn't able to respond or why the client had sent a partial request. Hopefully this will be sufficient to stop the failures.

In reply to @k900:0upti.me
https://hydra.nixos.org/build/202864292/nixlog/5