I’ve upgraded NixOS to a recent commit from unstable and ACME is not working anymore. All I see in the logs is:

Failed to start Renew ACME certificate for example.com.
acme-example.com.service: Failed to load environment files: No such file or directory
acme-example.com.service: Failed to run 'start' task: No such file or directory
acme-example.com.service: Failed with result 'resources'.

over and over again. (I replaced my actual domain with example.com)
I am using the cloudflare DNS challenge .

Does anyone have any pointers as to how I could debug this?

m1cr0man: so do you remember #153942? i didn't notice it at the time but the issue that it solved may be able to be made redundant.

https://github.com/NixOS/nixpkgs/commit/81a67a3353b09c0abade5f2d17e91d23873fc7fb added SupplementalGroups=acme if ACME certs are used to the Caddy service, which gives the Caddy service access to the certs mo matter what group the Caddy service user is a part of. (In fact, I think my assertions made it so you'd have to add the acme group to the caddy user, even if it would work fine without it due to SupplementalGroups, whoops.)

I think we can make this change across the board, and (potentially) remove the assertions? Let me know what you think.

Heyo 👋 Sorry got distracted and forgot to reply earlier. Heading off but I'll read any replies tomorrow.

Yeah this is interesting. SupplementalGroups certainly would raise false alarms with the assertion the way it is. When you say make the change across the board, what are you thinking of doing?

I'm also thinking that depending on your plans here that assuming the cert's group is acme wouldn't be sufficient and you'd want to rely on config.security.acme.certs.<name>.group in dependant services.

When you say make the change across the board, what are you thinking of doing?

Migrating all web servers that we support to use it instead of the assertions, ideally.

Yeah honestly I think that would be a good idea :) There will be some things to note however.

Firstly, we have weak values for group set on a cert used by nginx/httpd (example:
https://github.com/m1cr0man/nixpkgs/blob/674cfc91c7432662fc8ab96a6d17819f5517ddb8/nixos/modules/services/web-servers/nginx/default.nix#L967). It _might_ be necessary to check that the user/group for the web server isn't already in the cert's group, however knowing Systemd if you specify SupplementalGroups the user already has it'll probably no-op and be grand.

Secondly there was in the past some concern raised around granting acme group to other services because it would grant that service access to more certs than you may want. You might get some backlash in that regard. In reality, this is hard to operate around and for wildcard certs you're likely to only have 1 cert shared across multiple services anyway.

Lastly there was still some cases where people/services wanted root as the owner and before the useRoot option was added to acme, LoadCredential was the only solution here: https://github.com/NixOS/nixpkgs/pull/123261 (WOW just noticed this hasn't been merged). I bring this up because LoadCredential would also be a valid solution instead of SupplementalGroups, but because credentials are not re-read from disk when they change which is bad for ACME usage, I don't think it's preferred.

Riiight, completely forgot about that.

I think the best thing to do here is to revisit how the Caddy module operates in this regard -- so removing the blanket "acme" group addition. (Since I'm not sure the best way to do this, would it be appropriate to open an issue to discuss it with the module maintainer?)

In reply to @m1cr0man:m1cr0man.com
Yeah that's probably best, and so that it's on record on Github too

      group = mkOption {
        type = types.str;
        inherit (defaultAndText "group" "acme") default defaultText;
        description = "Group running the ACME client.";
      };

i feel like this description is inaccurate?

        # Group might change between runs, re-apply it
        chown '${user}:${data.group}' certificates/*

hm