!MthpOIxqJhTgrMNxDS:nixos.org

NixOS ACME / LetsEncrypt

105 Members
Another day, another cert renewal44 Servers

Load older messages


SenderMessageTime
9 Nov 2024
@k900:0upti.meK900 ⚡️https://github.com/NixOS/nixpkgs/pull/35462906:50:27
@k900:0upti.meK900 ⚡️ I added a commit that makes it work for now 06:50:34
@k900:0upti.meK900 ⚡️
webserver # [  426.884702] (es-start)[2816]: acme-lockfiles.service: Changing to the requested working directory failed: Permission denied
webserver # [  426.934208] (es-start)[2816]: acme-lockfiles.service: Failed at step CHDIR spawning /nix/store/n24xs3nmndyyivq3q5w52f7aqlb06hqh-unit-script-acme-lockfiles-start/bin/acme-lockfiles-start: Permission denied
08:03:11
@k900:0upti.meK900 ⚡️You fucking what08:03:13
@k900:0upti.meK900 ⚡️https://hydra.nixos.org/build/278094707/log08:05:12
@k900:0upti.meK900 ⚡️Also this thing08:05:13
@k900:0upti.meK900 ⚡️What is even happening anymore08:05:18
@k900:0upti.meK900 ⚡️OK looks like that machine is just hella overloaded08:06:32
@m1cr0man:m1cr0man.comm1cr0manLooking into that ^ The acme-lockfiles.service is configured in a less than stellar manner. Working directory is /run/acme, but it is managed by tmpfiles instead of RuntimeDirectory, despite being RemainAfterExit (so the runtime dir should not get deleted). Gonna fix all of this now.21:06:42
@k900:0upti.meK900 ⚡️https://hydra.nixos.org/eval/180987321:57:20
@k900:0upti.meK900 ⚡️More ordering nonsense 21:57:24
@k900:0upti.meK900 ⚡️If anyone wants to look into it 21:57:30
@k900:0upti.meK900 ⚡️It's funny how adding more synchronization uncovers more and more weird behaviors 21:58:31
@k900:0upti.meK900 ⚡️ Just because tests that used to insta-fail on slow machines don't anymore 21:58:43
@m1cr0man:m1cr0man.comm1cr0manDid I miss it? Looks like it passed22:20:42
@k900:0upti.meK900 ⚡️I restarted it 22:21:12
@k900:0upti.meK900 ⚡️Without really thinking 22:21:43
@k900:0upti.meK900 ⚡️My bad 22:21:44
@m1cr0man:m1cr0man.comm1cr0manNo worries. What was the gist of it?22:24:00
@k900:0upti.meK900 ⚡️Two different failures 22:24:33
@k900:0upti.meK900 ⚡️On aarch64 and x86_6422:24:41
@k900:0upti.meK900 ⚡️Did not look closely 22:24:44
@m1cr0man:m1cr0man.comm1cr0manok fair enough22:24:59
@k900:0upti.meK900 ⚡️I think it's just ordering being off again 22:28:34
@k900:0upti.meK900 ⚡️But I don't have a good mental model 22:28:39
@m1cr0man:m1cr0man.comm1cr0man Just looking over the maxConcurrentRenewals implementation, and the options/discussion from last year. I'm really starting to feel that the systemd dependency approach would have been more straightforward. I couldn't convince folks at the time, and there was lengthly discussion in which it really needed feedback from another maintainer. I went with the current solution because I felt there wasn't much between them and that it may lend to more ACME contributors, but I'm seeing now that it's a heavy bit of complexity and we're still limited on maintainers. 22:29:59
@m1cr0man:m1cr0man.comm1cr0man
In reply to @k900:0upti.me
Or the units need to also wants the server

An idea for fixing this: I could add more targets in the ACME module to simplify the config and dependencies in the webserver + other downstream modules, and potentially help resolve this issue also:

  • Add an acme-renewal-http01.target which requires and after the relevant acme services.
  • For each web server listening on port 80 or configured to serve the acme-challenge directory (either is possible and logic already exists to discover these cases), add a requires and before rule on acme-renewal-http01.target

Honestly, I'm trying to think of reasons I haven't done this until now. I could add targets for other renewal types with the intention to allow DNS server startups in the same way. I could even go further and add targets for acme-selfsigned.target and acme-renewal.target so that downstream services generally don't need to worry about what certs to wait on. I would hazard a guess that the complement of certs being waited on is significant in 90% of system configurations out there, and using these general targets wouldn't cause much more slow down.

23:09:17
@m1cr0man:m1cr0man.comm1cr0man
In reply to @k900:0upti.me
Or the units need to also wants the server
*

An idea for fixing this: I could add more targets in the ACME module to simplify the config and dependencies in the webserver + other downstream modules, and potentially help resolve this issue also:

  • Add an acme-renewal-http01.target which requires and after the relevant acme services.
  • For each web server listening on port 80 or configured to serve the acme-challenge directory (either is possible and logic already exists to discover these cases), add a requires and before rule on acme-renewal-http01.target

Honestly, I'm trying to think of reasons I haven't done this until now. I could add targets for other renewal types with the intention to allow DNS server startups in the same way. I could even go further and add targets for acme-selfsigned.target and acme-renewal.target so that downstream services generally don't need to worry about what certs to wait on. I would hazard a guess that the complement of certs being waited on is not significant in 90% of system configurations out there, and using these general targets wouldn't cause much more slow down.

23:23:15
10 Nov 2024
@m1cr0man:m1cr0man.comm1cr0man
In reply to @m1cr0man:m1cr0man.com
Just looking over the maxConcurrentRenewals implementation, and the options/discussion from last year. I'm really starting to feel that the systemd dependency approach would have been more straightforward. I couldn't convince folks at the time, and there was lengthly discussion in which it really needed feedback from another maintainer. I went with the current solution because I felt there wasn't much between them and that it may lend to more ACME contributors, but I'm seeing now that it's a heavy bit of complexity and we're still limited on maintainers.
Another thing about this - we already use systemd dependency ordering to do something very similar with how we handle account creation, where one cert is elected as a leader. It just feels unnecessary to have locks implemented on disk for this other use case.
00:08:54
@emilazy:matrix.orgemilyI forget what side I was on but we should go with that one 😂00:41:44

Show newer messages


Back to Room ListRoom Version: 6