| 17 May 2023 |
 hexa | reading through that ticket, I think that is exactly the same company actually | 22:45:33 |
| hexa | wondering if CPUSchedulingPolicy=idle would do the trick | 22:53:31 |
| 18 May 2023 |
 emily | In reply to @m1cr0man:m1cr0man.com
Thanks for jumping in there emily 🙂 I always feel a pressure to respond to these sort of issues (when someone is eager to open a PR) to make sure the module stays maintainable, it's good to know I'm not the only one looking out! no worries, sorry i'm barely around these days :) | 06:03:37 |
| emily | I was thinking CPUWeight=idle, didn't know about CPUSchedulingPolicy. I wonder what the difference is :/ | 06:04:31 |
| emily | I agree that the module is definitely plenty complex enough already (honestly too much in my opinion, I am quite down on lego at this point with how much it has required in the way of custom-built infrastructure and workarounds) | 06:06:05 |
 m1cr0man | Heh I'm barely around either so don't sweat it... I wish I had time to do more, such as opening PRs to Lego to upstream some features like offline renewal checking and proper handling of domain list changes. If they are resistant to those sort of things I'm not against beginning to migrate the module to another tool but... So many resources out there wrt setting up acme as well as people's configs rely on Lego. It's a bit vendor locked sadly | 08:17:58 |
| emily | yeah I feel like the last migration was painful enough that rewriting the whole thing yet again and probably breaking people's configurations in the process would hurt. | 08:34:38 |
| emily | I am more sympathetic to the Caddy "ACME should be integrated into the applications using the certs because there's a lot you can do more reliably/properly and certificate lifecycle management goes smoother" POV these days than I used to be, my personal plan was/is to switch to Caddy and have it dump out the cert to disk (which iirc it can do now) for non-HTTP things | 08:36:29 |
| emily | Caddy is explicitly advertised as being possible to use as a certificate management service separate from being used as a web server but I feel like migrating the whole module to anything else is hard to advocate for at this point unless lego seriously degrades somehow | 08:43:29 |
| emily | (Caddy itself used to use lego - indeed I think they are the reason lego exists? - and then abandoned it for their own implementation) | 08:44:16 |
| emily | btw I assume another effect of spamming the services on switch is that sometimes serves with tons of certificates will spam LE with (re)issuances all at once? that may be a stronger argument than CPU load for some kind of limited parallelism or randomized timing, because what about hosts with thousands of certificates? | 09:24:32 |
| emily | (e.g.: what if lots of hosts do an automated configuration switch at midnight/some other Schelling point for automatic nixpkgs upgrades and at scale this causes us to direct a bunch of predictable load to LE that the module already goes to pains to spread out during normal operation?) | 09:26:43 |
| hexa | In reply to @emilazy:matrix.org
I was thinking CPUWeight=idle, didn't know about CPUSchedulingPolicy. I wonder what the difference is :/ yeah, that was my concern as well. Too many options, that sound relevant. | 10:56:39 |
| 25 May 2023 |
|  raitobezarius changed their display name from raitobezarius to disko in NixOS 23.11 when. | 13:32:16 |
| raitobezarius changed their display name from disko in NixOS 23.11 when to raitobezarius. | 13:37:24 |
| 27 May 2023 |
|  mjolnirchanged room power levels. | 16:40:04 |
| 4 Jun 2023 |
| m1cr0man | Lol I had a dream last night that someone opened a critical ticket on the Acme module and a core maintainer had to fix it because I was asleep but when I read it, it was intended behaviour/user error and I was annoyed that the module got changed just for it 😂 I woke up and like panic checked my emails and of course there was nothing | 07:51:49 |
| 9 Jun 2023 |
| emily | despite all the headaches lego has caused I would like to spend a moment of thanks for the fact that we did not move to anything shell-based: https://github.com/acmesh-official/acme.sh/issues/4659 | 10:24:43 |
| emily | ah I see this already came up in the security channel | 11:19:04 |
| 13 Jun 2023 |
| m1cr0man | Hello again :) Busy few weeks... looking into https://github.com/NixOS/nixpkgs/issues/232505 again. I just had a notion - could we chain all the certs together with an After= condition? We would still need to avoid auto-starting the services for each cert (otherwise config switch would take a REALLY long time) but that might be easy to solve with a target. | 19:44:52 |
| m1cr0man | In reply to @emilazy:matrix.org
despite all the headaches lego has caused I would like to spend a moment of thanks for the fact that we did not move to anything shell-based: https://github.com/acmesh-official/acme.sh/issues/4659 Oh wow.. that's spooky. At least if we were using that our systemd services for renewal are hardened like steel | 19:45:33 |
| emily | only so much hardening you can do when the process has access to private keys :( | 20:03:36 |
| emily | (ideally you have privilege separation so that the process that talks to the ACME server doesn't have access to the keys but I don't think even lego does that) | 20:05:44 |
| emily | In reply to @m1cr0man:m1cr0man.com
Hello again :) Busy few weeks... looking into https://github.com/NixOS/nixpkgs/issues/232505 again. I just had a notion - could we chain all the certs together with an After= condition? We would still need to avoid auto-starting the services for each cert (otherwise config switch would take a REALLY long time) but that might be easy to solve with a target. honestly I don't know if there's a one-size-fits-all solution to this. we can randomize renewal time because it fundamentally doesn't matter when renewal happens as long as it's sufficiently far in advance. some users will want their sites accessible as soon as possible after setting up a new box or activating a new configuration; some will be worried about load and rate limits. i don't see how we can satisfy both out of the box | 20:06:49 |
| emily | the "This will cause the timer to start; and after 1 second start all the services with a randomised delay." idea sounds nice enough - but then we're talking about, your sites have broken SSL for up to an entire day? | 20:07:23 |
| emily | I'm curious how Caddy/certmagic handles this since it has pretty sophisticated logic for cert issue timing | 20:08:08 |
| m1cr0man | Could you let me know what you find from that? But to your point about one size fits all, it seems like we will need to introduce an option for users to decide what they want. We can default to the current situation, but provide an option like renewOnActivate for other situations? | 20:09:44 |
| emily | I'm tempted to say that people can just poke at the systemd.* options themselves if they really want rate limiting, but I'm biased :p | 20:10:27 |
| emily | I would consider it acceptable to do something out of the box if we found a solution that leads to large numbers of certs being activated in minutes rather than hours/days though | 20:10:48 |
| emily | if you have dozens/hundreds of certs then you're probably expecting initial setup to take about that long | 20:11:28 |
