In reply to @k900:0upti.me
Or the units need to also wants the server

An idea for fixing this: I could add more targets in the ACME module to simplify the config and dependencies in the webserver + other downstream modules, and potentially help resolve this issue also:

• Add an acme-renewal-http01.target which requires and after the relevant acme services.
• For each web server listening on port 80 or configured to serve the acme-challenge directory (either is possible and logic already exists to discover these cases), add a requires and before rule on acme-renewal-http01.target

Honestly, I'm trying to think of reasons I haven't done this until now. I could add targets for other renewal types with the intention to allow DNS server startups in the same way. I could even go further and add targets for acme-selfsigned.target and acme-renewal.target so that downstream services generally don't need to worry about what certs to wait on. I would hazard a guess that the complement of certs being waited on is significant in 90% of system configurations out there, and using these general targets wouldn't cause much more slow down.

In reply to @k900:0upti.me
Or the units need to also wants the server

An idea for fixing this: I could add more targets in the ACME module to simplify the config and dependencies in the webserver + other downstream modules, and potentially help resolve this issue also:

• Add an acme-renewal-http01.target which requires and after the relevant acme services.
• For each web server listening on port 80 or configured to serve the acme-challenge directory (either is possible and logic already exists to discover these cases), add a requires and before rule on acme-renewal-http01.target

Honestly, I'm trying to think of reasons I haven't done this until now. I could add targets for other renewal types with the intention to allow DNS server startups in the same way. I could even go further and add targets for acme-selfsigned.target and acme-renewal.target so that downstream services generally don't need to worry about what certs to wait on. I would hazard a guess that the complement of certs being waited on is not significant in 90% of system configurations out there, and using these general targets wouldn't cause much more slow down.

In reply to @m1cr0man:m1cr0man.com
Just looking over the maxConcurrentRenewals implementation, and the options/discussion from last year. I'm really starting to feel that the systemd dependency approach would have been more straightforward. I couldn't convince folks at the time, and there was lengthly discussion in which it really needed feedback from another maintainer. I went with the current solution because I felt there wasn't much between them and that it may lend to more ACME contributors, but I'm seeing now that it's a heavy bit of complexity and we're still limited on maintainers.

I don't see switching from lego being on the agenda for a good while.

I don't think anyone is planning to put in the work to make it happen, but I do think that we're very much at the point where our certificate management lifecycle just wants to be an autonomous always-running daemon that communicates with the rest of the system via systemd

Personally, I don't hate the fact that we've used systemd to achieve this. It is an always running daemon that communicates with the rest of the system 😉 and it integrates very nicely with the lifecycle of services which depend on acme certs. However we are definitely pushing (and actual exceeding) its limits in terms of what it can achieve.

As you said, it has been an organic evolution over many years for many use cases. I want to give refactoring one good go before investigating alternative solutions/replacement to the stack we have today. Perhaps there is something that would make life easier as maintainers, but from what feedback I've heard, people are generally happy with cert management today.

In reply to @m1cr0man:m1cr0man.com
If there's any maintainers about, I think this PR is good to merge also https://github.com/NixOS/nixpkgs/pull/348344

In reply to @m1cr0man:m1cr0man.com
https://github.com/NixOS/nixpkgs/pull/355087 first big refactoring PR

In reply to @k900:0upti.me
This scares me but in a good way

In reply to @k900:0upti.me
Merged that

This open, 2020 ticket is peak ACME module: https://github.com/NixOS/nixpkgs/issues/106862

This is actually a variant of what K900 saw yesterday wrt webserver startup ordering. It duplicates/affects this recent ticket too. I also found the reason why we check cert expiry ourselves (I recall there being an issue with container startup also? But I can't find any reference to it. Please link if you know of it.).

I see two ways to frame this issue more generally, each with very different solutions:

1. "ACME renewal does not reliably wait on external dependencies"

In this case, we need to have a reliable mechanism for configuring services which may affect cert renewability to be started before renewal is attempted. One solution is to add an acme-renewal-dependencies.target and add service modules to it as required on a best effort basis. I'm sure issues will be opened if we miss something, as they have been historically.

Sadly this only half-solves the problem. Running != listening, and systemd only accounts for the former in most cases (non-notify services). Socket units were suggested before (I do understand them now 😉) but that is a monumentous task for all the dependencies.
We could do some naive tests and delays like CURLing the configured ACME server and checking for a listening on port 80, but that just feels wrong.

1. "ACME renewal does not gracefully handle failures"

This is actually untrue - we do have systemd Restart directives configured on the units. The problem is that this causes start jobs to fail, and dependent services to not start or work. What we really want(ed) is a way to gracefully retry, where we don't fail the job. We could add some sort of retry logic to the script and do away with systemd's retry logic, but again feels like reinventing the systemd retry logic in a crude way.

I would love to reduce flakiness and remove scripts (the logic for checking renewal date) at the same time, but we're stuck with a limited toolset to solve this.

My feeling right now is that we should at least implement solution 1 and get dependency ordering right for non-failure scenarios. Despite the caveats, I still think this would be a significant improvement. I wish I could say with confidence that this would solve test flakiness, but it probably won't.