NixOS ACME / LetsEncrypt - Public Room Timeline

	NixOS ACME / LetsEncrypt	106 Members
	Another day, another cert renewal	47 Servers

Load older messages

Sender	Message	Time
29 Nov 2024
hexa	doesn't explain why `systemctl reload nginx` gets stuck 😄	18:58:45
hexa	Thread 1 (Thread 0x7f7f4c1c5680 (LWP 180909) "systemctl"): #0 0x00007f7f4c50963c in ppoll () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6 No symbol table info available. #1 0x00007f7f4c82270b in ppoll_usec () from target:/nix/store/ivqjhj99firnjq7gp14qf35821viwi5m-systemd-256.7/lib/systemd/libsystemd-shared-256.so No symbol table info available. #2 0x00007f7f4c89e33a in bus_poll () from target:/nix/store/ivqjhj99firnjq7gp14qf35821viwi5m-systemd-256.7/lib/systemd/libsystemd-shared-256.so No symbol table info available. #3 0x00007f7f4c89e6c5 in sd_bus_wait () from target:/nix/store/ivqjhj99firnjq7gp14qf35821viwi5m-systemd-256.7/lib/systemd/libsystemd-shared-256.so No symbol table info available. #4 0x00007f7f4c6c41b9 in bus_wait_for_jobs () from target:/nix/store/ivqjhj99firnjq7gp14qf35821viwi5m-systemd-256.7/lib/systemd/libsystemd-shared-256.so No symbol table info available. #5 0x00005641d8690e2c in verb_start () No symbol table info available. #6 0x00005641d8672bea in main ()	18:59:54
ThinkChaos	You could run the service's `ExecReload` manually to see if it's there or in Systemd it's hanging	19:11:11
ThinkChaos	It only does 2 things: check the config, and send a SIGHUP	19:11:58
hexa	systemctl reload nginx blocks, I think I established that earlier	19:16:32
hexa	uhh, sorry	19:16:37
hexa	I mean I established that the both work individually	19:16:56
hexa	it is systemctl reload that is stuck for some reason	19:17:03
hexa	https://gist.github.com/mweinelt/f099ec270ace7cb197954e23871471be	19:21:08
	@admin:nixos.org joined the room.	19:22:24
	@admin:nixos.org left the room.	19:22:37
ThinkChaos	Respectfully, I don't want to spend more time investigating this issue since it's in your personal config and not the NixOS modules. Your strace ends with `ask-password` related stuff so it's likely waiting to authenticate somehow. If you switch to `reloadServices` it uses `--no-block`. And better yet, if you switch to `enableReload` you'll use the battle tested solution.	19:54:56
1 Dec 2024
m1cr0man	I have another "fun" set of upstreaming work completed. I estimate this one at half the chance of being merged than the previous change, simply because of the structure of lego's cmd code + error handling. https://github.com/go-acme/lego/compare/master...m1cr0man:lego:renew-rc-2 https://github.com/m1cr0man/nixpkgs/commit/53846b07f5037e854993366beab3e0a618d1fd68 I have not opened PRs yet, will do that in a second	01:52:09
m1cr0man	With this work, I think the ACME module is in one of the best states it has ever been in. The remaining bash scripting in the module does only 2 things primarily: 1. Perform simple file operations like cp, chmod, chown. 2. Handle concurrency limits. The latter is being looked into by ThinkChaos too, see earlier discussions :)	02:00:32
m1cr0man	Lol, that ended quickly https://github.com/go-acme/lego/pull/2366	02:18:10
m1cr0man	https://github.com/go-acme/lego/issues/2367 🤷 lets hope it doesn't take years	02:37:50
5 Dec 2024
	maka_77x joined the room.	01:53:01
16 Dec 2024
K900 (Old)	So uh	23:41:24
K900 (Old)	Do we have anything that can at least paper over the ordering issues	23:41:42
K900 (Old)	Without making things even more complicated	23:41:51
K900 (Old)	Because the tests are flaking a lot and it's getting on my nerves	23:42:04
17 Dec 2024
ThinkChaos	Could someone please review the fix for cert ownership error message causing an unrelated exception PR, #362271? It's a tiny diff :) Users are getting misleading errors due to this throwing ATM	23:57:19
19 Dec 2024
ThinkChaos	K900 I looked at the log for the this failure, httpd only started after the ACME validation happened: `Starting Apache HTTPD` vs `Attempting to validate w/ HTTP` I think this is a `switch-to-configuration-ng` regression 😕 The perl script starts all services in a single systemctl call, so a single Systemd transaction. That means httpd's Before relationship with the certs is enforced. Whereas -ng uses the Systemd D-BUS API to start services one by one, meaning multiple transactions. So Before is not enforced. I guess we can try and disable -ng for the ACME tests, see how it goes for a week or so and then potentially raise an issue with -ng.	01:31:18
ThinkChaos	BTW thanks for the review + merge on the PR from above!	01:39:02
K900 (Old)	In reply to @thinkchaos:matrix.org K900 I looked at the log for the this failure, httpd only started after the ACME validation happened: `Starting Apache HTTPD` vs `Attempting to validate w/ HTTP` I think this is a `switch-to-configuration-ng` regression 😕 The perl script starts all services in a single systemctl call, so a single Systemd transaction. That means httpd's Before relationship with the certs is enforced. Whereas -ng uses the Systemd D-BUS API to start services one by one, meaning multiple transactions. So Before is not enforced. I guess we can try and disable -ng for the ACME tests, see how it goes for a week or so and then potentially raise an issue with -ng. Uhh	06:55:59
K900 (Old)	Can you please report this in #NixOS systemd	06:56:24
Arian	There is no api for starting multiple services in a single transaction. This has always been a lie	10:46:30
Arian	I think systemctl start also is a for loop around starting single units through dbus afaicr	10:46:51
ThinkChaos	Yeah I need to dig a bit more before I make too much noise, I'll look at systemctl's code, thanks for the hint	13:38:17
ThinkChaos	Either way I think we'll need to make the link between the certs and web server stronger to fix this: I'm thinking certs using HTTP validation can `Require` the relevant web server	13:45:07

Show newer messages

Back to Room ListRoom Version: 6