Personally, I don't hate the fact that we've used systemd to achieve this. It is an always running daemon that communicates with the rest of the system 😉 and it integrates very nicely with the lifecycle of services which depend on acme certs. However we are definitely pushing (and actual exceeding) its limits in terms of what it can achieve.

As you said, it has been an organic evolution over many years for many use cases. I want to give refactoring one good go before investigating alternative solutions/replacement to the stack we have today. Perhaps there is something that would make life easier as maintainers, but from what feedback I've heard, people are generally happy with cert management today.

In reply to @m1cr0man:m1cr0man.com
If there's any maintainers about, I think this PR is good to merge also https://github.com/NixOS/nixpkgs/pull/348344

In reply to @m1cr0man:m1cr0man.com
https://github.com/NixOS/nixpkgs/pull/355087 first big refactoring PR

In reply to @k900:0upti.me
This scares me but in a good way

In reply to @k900:0upti.me
Merged that

This open, 2020 ticket is peak ACME module: https://github.com/NixOS/nixpkgs/issues/106862

This is actually a variant of what K900 saw yesterday wrt webserver startup ordering. It duplicates/affects this recent ticket too. I also found the reason why we check cert expiry ourselves (I recall there being an issue with container startup also? But I can't find any reference to it. Please link if you know of it.).

I see two ways to frame this issue more generally, each with very different solutions:

1. "ACME renewal does not reliably wait on external dependencies"

In this case, we need to have a reliable mechanism for configuring services which may affect cert renewability to be started before renewal is attempted. One solution is to add an acme-renewal-dependencies.target and add service modules to it as required on a best effort basis. I'm sure issues will be opened if we miss something, as they have been historically.

Sadly this only half-solves the problem. Running != listening, and systemd only accounts for the former in most cases (non-notify services). Socket units were suggested before (I do understand them now 😉) but that is a monumentous task for all the dependencies.
We could do some naive tests and delays like CURLing the configured ACME server and checking for a listening on port 80, but that just feels wrong.

1. "ACME renewal does not gracefully handle failures"

This is actually untrue - we do have systemd Restart directives configured on the units. The problem is that this causes start jobs to fail, and dependent services to not start or work. What we really want(ed) is a way to gracefully retry, where we don't fail the job. We could add some sort of retry logic to the script and do away with systemd's retry logic, but again feels like reinventing the systemd retry logic in a crude way.

I would love to reduce flakiness and remove scripts (the logic for checking renewal date) at the same time, but we're stuck with a limited toolset to solve this.

My feeling right now is that we should at least implement solution 1 and get dependency ordering right for non-failure scenarios. Despite the caveats, I still think this would be a significant improvement. I wish I could say with confidence that this would solve test flakiness, but it probably won't.