!MthpOIxqJhTgrMNxDS:nixos.org

NixOS ACME / LetsEncrypt

105 Members
Another day, another cert renewal43 Servers

Load older messages

SenderMessageTime
19 Apr 2025
@hexa:lossy.networkhexa 
2025/04/19 22:39:09 [INFO] [music.lossy.network] acme: renewalInfo endpoint indicates that renewal is needed
2025/04/19 22:39:09 [INFO] [music.lossy.network] Sleeping 21h43m27.656213001s until renewal time 2025-04-20 20:22:37.463135258 +0000 UTC
22:49:56
@hexa:lossy.networkhexabut that resulted in nginx not starting up22:50:03
@hexa:lossy.networkhexabecause it depends on all the acme-${domain}.service units22:50:28
@emilazy:matrix.orgemilyhm, I thought we were going to set it to just not wait?22:52:10
@hexa:lossy.networkhexaand we did not set it to anything in nixpkgs22:54:08
@hexa:lossy.networkhexabut I set it to something on my private infra22:54:16
@emilazy:matrix.orgemilyright23:00:12
@emilazy:matrix.orgemilyI think the current format will only work well when set to not wait at all23:00:19
@emilazy:matrix.orgemily(which should be fine as the cron job runs often anyway, though we might want to bump it)23:00:29
21 Apr 2025
@m1cr0man:m1cr0man.comm1cr0manThere was some talk about bumping it when they announced the lower lifetime certs. Wouldn't be the worst thing to do.19:18:58
22 Apr 2025
@hexa:lossy.networkhexanow 47 days was announced to be the next shorter lifespan23:08:50
@hexa:lossy.networkhexaand I don't think it warrants trying more than daily for 7-14 days23:09:13
@hexa:lossy.networkhexa* and I don't think it warrants trying more than daily23:09:33
@hexa:lossy.networkhexafor 6 days that changes of course23:09:45
28 Apr 2025
@m1cr0man:m1cr0man.comm1cr0man https://github.com/NixOS/nixpkgs/pull/376334#pullrequestreview-2801003367 this is ready to go. I tested it too. 21:26:09
29 Apr 2025
@ygt:matrix.orgnadir left the room.23:42:45
5 May 2025
@netpleb:matrix.orgnetpleb

hi everyone, does anybody have a workaround that fixes this pesky dns resolution issue when acme.certs... and BIND are running in a declarative nixos container?

Could not create client: get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get "https://acme-v02.api.letsencrypt.org/directory": GET https://acme-v02.api.letsencrypt.org/directory giving up after 6 attempt(s): Get "https://acme-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-v02.api.letsencrypt.org: Temporary failure in name resolution
17:59:16
@netpleb:matrix.orgnetplebwhat seems to be happening is that acme is starting so early that the container is unable to route things yet. Maybe the host has not installed routes yet? but that somehow acme blocks until it times out.18:00:34
@netpleb:matrix.orgnetplebonce it times out, then everything works fine. I have whittled it down to acme, because when I remove any acme things the container boots up just fine and is able to route/ping quite quickly18:01:46
@netpleb:matrix.orgnetpleb

so, by that, I mean that the issue does not seem to be pertaining to (in my case) networking.wireguard nor networking.bind which are both used and operate completely fine in the container. It is only after adding something like (per the manual):

      security.acme.certs."<redacted>"= {
        domain = "*<redacted>";
        dnsProvider = "rfc2136"; # allows us to do dns acme validation with local dns server
        environmentFile = "/var/lib/secrets/<redacted>.certs.secret";
        # We don't need to wait for propagation since this is a local DNS server
        dnsPropagationCheck = false;
      };

that the behavior occurs.

18:04:57
@netpleb:matrix.orgnetpleb what I am most confused about (and why I am posting here) is why the call to lego --accept-tos --path . -d '*.<redacted>' --email <redacted> --key-type ec256 --dns rfc2136 --dns.propagation-disable-ans --dns.resolvers 127.0.0.1:53 --server https://acme-v02.api.letsencrypt.org/directory renew --no-random-sleep --days 30 seems to block all network traffic, even for other services (like wireguard, bind, etc) 18:07:29
@netpleb:matrix.orgnetpleb * what I am most confused about (and why I am posting here) is why the call to lego --accept-tos --path . -d '*.<redacted>' --email <redacted> --key-type ec256 --dns rfc2136 --dns.propagation-disable-ans --dns.resolvers 127.0.0.1:53 --server https://acme-v02.api.letsencrypt.org/directory renew --no-random-sleep --days 30 seems to block all network traffic, even for other services (like wireguard, bind, etc) until it times out. There must be something I do not understand about how systemd works or calls this, but I woudl like to learn ;) 18:07:56
@netpleb:matrix.orgnetpleb * what I am most confused about (and why I am posting here) is why the call to lego --accept-tos --path . -d '*.<redacted>' --email <redacted> --key-type ec256 --dns rfc2136 --dns.propagation-disable-ans --dns.resolvers 127.0.0.1:53 --server https://acme-v02.api.letsencrypt.org/directory renew --no-random-sleep --days 30 seems to block all network traffic, even for other services (like wireguard, bind, etc) until it times out. There must be something I do not understand about how systemd works or calls this, but I would like to learn ;) 18:08:04
@netpleb:matrix.orgnetpleb in essence though, as soon as I comment out the security.acme.certs... config above, the container boots up in a couple seconds, whereas with the acme config in place it takes a couple minutes since it has to wait for acme to timeout. I have tried for days now to figure out how to move the acme renewal process way later, but nothing seems to work. 18:34:26
@netpleb:matrix.orgnetpleb * in essence though, as soon as I comment out the security.acme.certs... config above, the container boots up in a couple seconds and can ping various ips and even resolve hostnames with the local BIND instance, whereas with the acme config in place it takes a couple minutes since it has to wait for acme to timeout. In the interim no pinging or hostname lookups even work. I have tried for days now to figure out how to move the acme renewal process way later, but nothing seems to work. 18:35:53
@netpleb:matrix.orgnetpleb * in essence though, as soon as I comment out the security.acme.certs... config above, the container boots up in a couple seconds and can ping various ips and even resolve hostnames with the local BIND instance, whereas with the acme config in place it takes a couple minutes to boot since it has to wait for acme to timeout. In the interim no pinging or hostname lookups even work. I have tried for days now to figure out how to move the acme renewal process way later, but nothing seems to work. 18:48:20
@netpleb:matrix.orgnetpleb (sorry for so many messages), I have continued to investigate and it seems that the root cause is that the host machine does not provide the network/routes to the container until late (possibly even after?) the container is done booting. So because of this, acme stalls the boot process. So far the only thing that has sort of worked, but is very not-clean, is for me to just put serviceConfig.TimeoutStartSec = "20s"; on the various acme-<domain>.service units. 20:18:57
6 May 2025
@m1cr0man:m1cr0man.comm1cr0manSorry - only seeing your messages now. I believe a fix for this does exist in the wild, I vaguely remember running into it a few years ago. Let me do some digging20:36:18
@m1cr0man:m1cr0man.comm1cr0man

In the mean time netpleb - can you provide the following info from within the container:

  • Logs of acme-$cert.service redacted as necessary
  • Output of systemctl list-dependencies acme-$cert.service
  • Output of systemctl list-dependencies bind.service
20:40:38
@m1cr0man:m1cr0man.comm1cr0man Ah, I see you already found the relevant ticket on GitHub. Did you try this fix? 20:42:27

Show newer messages

Back to Room ListRoom Version: 6