| 7 Feb 2024 |
 K900 | That's not really a NixOS problem | 22:01:23 |
| K900 | More of a general networking problem | 22:01:28 |
 netpleb | ok, sorry, i fixed the dns issue already. I am now able to ping it:
[root@netpleb-public-services:~]# ping acme-v02.api.letsencrypt.org
PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248) 56(84) bytes of data.
64 bytes from 172.65.32.248: icmp_seq=1 ttl=59 time=94.5 ms
64 bytes from 172.65.32.248: icmp_seq=2 ttl=59 time=93.1 ms
64 bytes from 172.65.32.248: icmp_seq=3 ttl=59 time=108 ms
64 bytes from 172.65.32.248: icmp_seq=4 ttl=59 time=100 ms
| 22:02:04 |
| K900 | Now you can restart the ACME service | 22:03:14 |
| K900 | And maybe it'll actually succeed | 22:03:18 |
| netpleb | In reply to @k900:0upti.me
Now you can restart the ACME service ok, is there a "parent" acme service i should restart that will redo all of them? i have one for a subdomain and one for the tld | 22:04:03 |
| K900 | No | 22:04:24 |
| netpleb | [root@netpleb-public-services:~]# systemctl restart acme-netpleb.com.service
A dependency job for acme-netpleb.com.service failed. See 'journalctl -xe' for details.
[root@netpleb-public-services:~]# journalctl -xeu acme-netpleb.com.service
Feb 07 21:59:35 netpleb-public-services systemd[1]: Dependency failed for Renew ACME certificate for netpleb.com.
░░ Subject: A start job for unit acme-netpleb.com.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit acme-netpleb.com.service has finished with a failure.
░░
░░ The job identifier is 73 and the job result is dependency.
Feb 07 21:59:35 netpleb-public-services systemd[1]: acme-netpleb.com.service: Job acme-netpleb.com.service/start failed with result 'dependency'.
| 22:05:20 |
| netpleb | the subdomain got further along it seems:
[root@netpleb-public-services:~]# systemctl status acme-jitsi.netpleb.com
× acme-jitsi.netpleb.com.service - Renew ACME certificate for jitsi.netpleb.com
Loaded: loaded (/etc/systemd/system/acme-jitsi.netpleb.com.service; linked; preset: enabled)
Active: failed (Result: exit-code) since Wed 2024-02-07 22:06:08 UTC; 17s ago
TriggeredBy: ● acme-jitsi.netpleb.com.timer
Process: 1244 ExecStart=/nix/store/miwhrhajjh9n1pz8zlb5vywnl6qczfad-unit-script-acme-jitsi.netpleb.com-start/bin/acme-jitsi.netpleb.com-start (code=exited, status=10)
Main PID: 1244 (code=exited, status=10)
CPU: 94ms
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1247]: 2024/02/07 22:06:08 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/312750532087
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1247]: 2024/02/07 22:06:08 Could not obtain certificates:
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1247]: error: one or more domains had a problem:
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1247]: [jitsi.netpleb.com] [jitsi.netpleb.com] acme: error presenting token: rfc2136: failed to insert: DNS update failed: server replied: SERVFAIL
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1244]: + echo Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1244]: Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1244]: + exit 10
Feb 07 22:06:08 netpleb-public-services systemd[1]: acme-jitsi.netpleb.com.service: Main process exited, code=exited, status=10/n/a
Feb 07 22:06:08 netpleb-public-services systemd[1]: acme-jitsi.netpleb.com.service: Failed with result 'exit-code'.
Feb 07 22:06:08 netpleb-public-services systemd[1]: Failed to start Renew ACME certificate for jitsi.netpleb.com.
| 22:07:52 |
| netpleb | * the subdomain got further along it seems (also, thank you in advance for your help, I have been struggling with this for days before reaching out here):
[root@netpleb-public-services:~]# systemctl status acme-jitsi.netpleb.com
× acme-jitsi.netpleb.com.service - Renew ACME certificate for jitsi.netpleb.com
Loaded: loaded (/etc/systemd/system/acme-jitsi.netpleb.com.service; linked; preset: enabled)
Active: failed (Result: exit-code) since Wed 2024-02-07 22:06:08 UTC; 17s ago
TriggeredBy: ● acme-jitsi.netpleb.com.timer
Process: 1244 ExecStart=/nix/store/miwhrhajjh9n1pz8zlb5vywnl6qczfad-unit-script-acme-jitsi.netpleb.com-start/bin/acme-jitsi.netpleb.com-start (code=exited, status=10)
Main PID: 1244 (code=exited, status=10)
CPU: 94ms
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1247]: 2024/02/07 22:06:08 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/312750532087
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1247]: 2024/02/07 22:06:08 Could not obtain certificates:
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1247]: error: one or more domains had a problem:
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1247]: [jitsi.netpleb.com] [jitsi.netpleb.com] acme: error presenting token: rfc2136: failed to insert: DNS update failed: server replied: SERVFAIL
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1244]: + echo Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1244]: Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1244]: + exit 10
Feb 07 22:06:08 netpleb-public-services systemd[1]: acme-jitsi.netpleb.com.service: Main process exited, code=exited, status=10/n/a
Feb 07 22:06:08 netpleb-public-services systemd[1]: acme-jitsi.netpleb.com.service: Failed with result 'exit-code'.
Feb 07 22:06:08 netpleb-public-services systemd[1]: Failed to start Renew ACME certificate for jitsi.netpleb.com.
| 22:08:25 |
| K900 | Your DNS server said no | 22:10:12 |
| K900 | You should now go look at the logs for that | 22:10:24 |
| netpleb | In reply to @k900:0upti.me
You should now go look at the logs for that ok. Progress finally! See this:
Feb 07 22:01:03 netpleb-public-services named[524]: client @0x7f186e0f4768 127.0.0.1#40626/key rfc2136key.netpleb.com: signer "rfc2136key.netpleb.com" approved
Feb 07 22:01:03 netpleb-public-services named[524]: client @0x7f186e0f4768 127.0.0.1#40626/key rfc2136key.netpleb.com: updating zone 'netpleb.com/IN': deleting rrset at '_acme-challenge.jitsi.netpleb.com' TXT
Feb 07 22:01:03 netpleb-public-services named[524]: client @0x7f186e0f4768 127.0.0.1#40626/key rfc2136key.netpleb.com: updating zone 'netpleb.com/IN': adding an RR at '_acme-challenge.jitsi.netpleb.com' TXT "JMV6KVjVQtGlCFKSucMcbbCN8RqGY9_ZBZC3sVr9NW0"
Feb 07 22:01:03 netpleb-public-services named[524]: client @0x7f186e0f4768 127.0.0.1#40626/key rfc2136key.netpleb.com: updating zone 'netpleb.com/IN': error: journal open failed: unexpected error
Feb 07 22:01:03 netpleb-public-services named[524]: client @0x7f186c582368 127.0.0.1#50260/key rfc2136key.netpleb.com: signer "rfc2136key.netpleb.com" approved
| 22:13:24 |
| netpleb | it is unclear to me which journal it is talking about. I also did this setup verbatim from the nixos manual, but originally I did have NSD installed. Maybe that is related? | 22:20:02 |
| netpleb | * it is unclear to me which journal it is talking about? | 22:51:26 |
| netpleb | how do you guys install your zone? do you use something like environment.etc."bind/zones/example.com.zone".source = ./example.com.zone where ./example.com.zone is in the git repo (I am using flakes) | 23:29:36 |
| netpleb | * how do you guys install your zone? do you use something like environment.etc."bind/zones/example.com.zone".source = ./example.com.zone; where ./example.com.zone is in the git repo (I am using flakes) | 23:29:50 |
| 8 Feb 2024 |
| netpleb | finally fixed it...had to make a oneshot service that chmod --recursive named:named /etc/bind/zones so that named has permission to load some .jnl file which apparently it needs to do the acme stuff. | 01:09:16 |
|  symys joined the room. | 18:51:42 |
| netpleb | to round this "issue" out for anyone else who comes along trying to figure out something similar: it turns out that when you follow the "fully self-hosted example using bind" in the manual, there is a subtle thing which probably goes unnoticed for many, namely
bind.zones.*.file = "/var/db/bind/${name}"; is in a directory which needs to be read/writable by bind. On my machine that directory had not yet even been created, and never actually was. This is because I had instead set bind.zones.*.file = ./zone-file-in-my-git-repo which means that Nix put the zone file into the nix store (which is fine as far as I am concerned) but the problem is that bind tries to create the .jnl file right next to it when doing the acme updates, which it obviously cannot do. So that is what threw the permission denied error which took a while to track down. | 18:56:44 |
| netpleb | so, in the end, what I am wondering is whether this is actually an issue with the bind package and module, and not the secruity.acme module. What do you guys think? | 18:57:22 |
| netpleb | * to round this "issue" out for anyone else who comes along trying to figure out something similar: it turns out that when you follow the "fully self-hosted example using bind" in the manual, there is a subtle thing which probably goes unnoticed for many, namely
bind.zones.*.file = "/var/db/bind/${name}"; is in a directory which needs to be read/writable by bind. On my machine that directory had not yet even been created, and never actually was. This is because I had instead set bind.zones.*.file = ./zone-file-in-my-git-repo which means that Nix put the zone file into the nix store (which is fine as far as I am concerned) but the problem is that bind tries to create the .jnl file right next to it when doing the acme updates, which it obviously cannot do. So that is what threw the permission denied error which took a while to track down. (and for which I am grateful to K900 ⚡️ for your patience in helping me get there!) | 18:58:44 |
| netpleb | * so, in the end, what I am wondering is whether this is actually an issue with the bind package and module, and not the secruity.acme module. What do you guys think? or maybe it is just a documentation thing? | 18:59:11 |
| netpleb | * to round this "issue" out for anyone else who comes along trying to figure out something similar: it turns out that when you follow the "fully self-hosted example using bind" in the manual, there is a subtle thing which probably goes unnoticed for many, namely
bind.zones.*.file = "/var/db/bind/${name}"; is in a directory which needs to be read/writable by bind. On my machine that directory had not yet even been created, and never actually was. This is because I had instead set bind.zones.*.file = ./zone-file-in-my-git-repo which means that Nix put the zone file into the nix store (which is fine as far as I am concerned) but the problem is that bind tries to create the .jnl file right next to it (in the nix store) when doing the acme updates, which it obviously cannot/should not do. So that is what threw the permission denied error which took a while to track down. (and for which I am grateful to K900 ⚡️ for your patience in helping me get there!) | 19:00:25 |
| netpleb | * to round this "issue" out for anyone else who comes along trying to figure out something similar: it turns out that when you follow the "fully self-hosted example using bind" in the manual, there is a subtle thing which probably goes unnoticed for many, namely
bind.zones.*.file = "/var/db/bind/${name}"; is in a directory which needs to be read/writable by bind. On my machine that directory had not yet even been created, and never actually was. This is because I had instead set bind.zones.*.file = ./zone-file-in-my-git-repo which means that Nix put the zone file into the nix store (which is fine as far as I am concerned) but the problem is that bind tries to create the .jnl file right next to it (in the nix store) when doing the acme updates, which it obviously cannot/should not do. So that is what threw the permission denied error which took a while to track down. (and for which I am grateful to K900 ⚡️ for your patience in helping me get there!)
My probably-not-the-best workaround was to add the zone file the the machine using something like environment.etc."bind/zones/the-zone-file"; and then a oneshot systemd service which runs the chmod --recursive named:named /etc/bind/zones command. Then everything seemed to finally work.
| 19:09:47 |
| 18 Feb 2024 |
|  link2xt joined the room. | 02:44:10 |
| 21 Feb 2024 |
|  @pederbs:pvv.ntnu.no left the room. | 00:50:58 |
| 6 Mar 2024 |
|  @sammy:cherrykitten.dev joined the room. | 19:03:45 |
| 7 Mar 2024 |
 hexa | acme test failure on unstable-small https://gist.github.com/mweinelt/7398a4d24ef4a4cd8f9dfa771ecb1f2a | 02:12:57 |
| hexa |
Test "Can request certificate with Lego's built in web server" failed with error: "unit "acme-finished-http.example.test.target" is inactive and there are no pending jobs"
| 02:13:15 |
