| 7 Feb 2024 |
 netpleb | [root@netpleb-public-services:~]# systemctl status acme-netpleb.com.service
○ acme-netpleb.com.service - Renew ACME certificate for netpleb.com
Loaded: loaded (/etc/systemd/system/acme-netpleb.com.service; linked; preset: enabled)
Active: inactive (dead)
TriggeredBy: ● acme-netpleb.com.timer
Feb 07 21:48:41 netpleb-public-services systemd[1]: Dependency failed for Renew ACME certificate for netpleb.com.
Feb 07 21:48:41 netpleb-public-services systemd[1]: acme-netpleb.com.service: Job acme-netpleb.com.service/start failed with result 'dependency'.
[root@netpleb-public-services:~]# ping netpleb.com
PING netpleb.com (38.45.103.128) 56(84) bytes of data.
64 bytes from ns1.netpleb.com (38.45.103.128): icmp_seq=1 ttl=64 time=0.041 ms
64 bytes from ns1.netpleb.com (38.45.103.128): icmp_seq=2 ttl=64 time=0.064 ms
^C
--- netpleb.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1064ms
rtt min/avg/max/mdev = 0.041/0.052/0.064/0.011 ms
| 21:51:06 |
 K900 | Not that | 21:51:36 |
| K900 | acme-v02.api.letsencrypt.org | 21:51:46 |
| K900 | Can you ping that? | 21:51:49 |
| netpleb | hmm, nope! wtf, I can ping google.com just fine though. What is going on? | 21:53:15 |
| K900 | You have a DNS problem | 21:55:01 |
| K900 | Have fun | 21:55:03 |
| netpleb | I am obviously not an expert in these things (though getting to know/learn Nix, both the language and the OS has been overall a rewarding experience). How is it possible that I can ping google but not letsencrypt? | 21:58:24 |
| netpleb | * ... | 21:59:26 |
| K900 | Something about your DNS config is broken | 22:01:14 |
| K900 | That's not really a NixOS problem | 22:01:23 |
| K900 | More of a general networking problem | 22:01:28 |
| netpleb | ok, sorry, i fixed the dns issue already. I am now able to ping it:
[root@netpleb-public-services:~]# ping acme-v02.api.letsencrypt.org
PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248) 56(84) bytes of data.
64 bytes from 172.65.32.248: icmp_seq=1 ttl=59 time=94.5 ms
64 bytes from 172.65.32.248: icmp_seq=2 ttl=59 time=93.1 ms
64 bytes from 172.65.32.248: icmp_seq=3 ttl=59 time=108 ms
64 bytes from 172.65.32.248: icmp_seq=4 ttl=59 time=100 ms
| 22:02:04 |
| K900 | Now you can restart the ACME service | 22:03:14 |
| K900 | And maybe it'll actually succeed | 22:03:18 |
| netpleb | In reply to @k900:0upti.me
Now you can restart the ACME service ok, is there a "parent" acme service i should restart that will redo all of them? i have one for a subdomain and one for the tld | 22:04:03 |
| K900 | No | 22:04:24 |
| netpleb | [root@netpleb-public-services:~]# systemctl restart acme-netpleb.com.service
A dependency job for acme-netpleb.com.service failed. See 'journalctl -xe' for details.
[root@netpleb-public-services:~]# journalctl -xeu acme-netpleb.com.service
Feb 07 21:59:35 netpleb-public-services systemd[1]: Dependency failed for Renew ACME certificate for netpleb.com.
░░ Subject: A start job for unit acme-netpleb.com.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit acme-netpleb.com.service has finished with a failure.
░░
░░ The job identifier is 73 and the job result is dependency.
Feb 07 21:59:35 netpleb-public-services systemd[1]: acme-netpleb.com.service: Job acme-netpleb.com.service/start failed with result 'dependency'.
| 22:05:20 |
| netpleb | the subdomain got further along it seems:
[root@netpleb-public-services:~]# systemctl status acme-jitsi.netpleb.com
× acme-jitsi.netpleb.com.service - Renew ACME certificate for jitsi.netpleb.com
Loaded: loaded (/etc/systemd/system/acme-jitsi.netpleb.com.service; linked; preset: enabled)
Active: failed (Result: exit-code) since Wed 2024-02-07 22:06:08 UTC; 17s ago
TriggeredBy: ● acme-jitsi.netpleb.com.timer
Process: 1244 ExecStart=/nix/store/miwhrhajjh9n1pz8zlb5vywnl6qczfad-unit-script-acme-jitsi.netpleb.com-start/bin/acme-jitsi.netpleb.com-start (code=exited, status=10)
Main PID: 1244 (code=exited, status=10)
CPU: 94ms
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1247]: 2024/02/07 22:06:08 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/312750532087
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1247]: 2024/02/07 22:06:08 Could not obtain certificates:
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1247]: error: one or more domains had a problem:
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1247]: [jitsi.netpleb.com] [jitsi.netpleb.com] acme: error presenting token: rfc2136: failed to insert: DNS update failed: server replied: SERVFAIL
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1244]: + echo Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1244]: Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1244]: + exit 10
Feb 07 22:06:08 netpleb-public-services systemd[1]: acme-jitsi.netpleb.com.service: Main process exited, code=exited, status=10/n/a
Feb 07 22:06:08 netpleb-public-services systemd[1]: acme-jitsi.netpleb.com.service: Failed with result 'exit-code'.
Feb 07 22:06:08 netpleb-public-services systemd[1]: Failed to start Renew ACME certificate for jitsi.netpleb.com.
| 22:07:52 |
| netpleb | * the subdomain got further along it seems (also, thank you in advance for your help, I have been struggling with this for days before reaching out here):
[root@netpleb-public-services:~]# systemctl status acme-jitsi.netpleb.com
× acme-jitsi.netpleb.com.service - Renew ACME certificate for jitsi.netpleb.com
Loaded: loaded (/etc/systemd/system/acme-jitsi.netpleb.com.service; linked; preset: enabled)
Active: failed (Result: exit-code) since Wed 2024-02-07 22:06:08 UTC; 17s ago
TriggeredBy: ● acme-jitsi.netpleb.com.timer
Process: 1244 ExecStart=/nix/store/miwhrhajjh9n1pz8zlb5vywnl6qczfad-unit-script-acme-jitsi.netpleb.com-start/bin/acme-jitsi.netpleb.com-start (code=exited, status=10)
Main PID: 1244 (code=exited, status=10)
CPU: 94ms
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1247]: 2024/02/07 22:06:08 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/312750532087
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1247]: 2024/02/07 22:06:08 Could not obtain certificates:
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1247]: error: one or more domains had a problem:
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1247]: [jitsi.netpleb.com] [jitsi.netpleb.com] acme: error presenting token: rfc2136: failed to insert: DNS update failed: server replied: SERVFAIL
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1244]: + echo Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1244]: Failed to fetch certificates. This may mean your DNS records are set up incorrectly. Selfsigned certs are in place and dependant services will still start.
Feb 07 22:06:08 netpleb-public-services acme-jitsi.netpleb.com-start[1244]: + exit 10
Feb 07 22:06:08 netpleb-public-services systemd[1]: acme-jitsi.netpleb.com.service: Main process exited, code=exited, status=10/n/a
Feb 07 22:06:08 netpleb-public-services systemd[1]: acme-jitsi.netpleb.com.service: Failed with result 'exit-code'.
Feb 07 22:06:08 netpleb-public-services systemd[1]: Failed to start Renew ACME certificate for jitsi.netpleb.com.
| 22:08:25 |
| K900 | Your DNS server said no | 22:10:12 |
| K900 | You should now go look at the logs for that | 22:10:24 |
| netpleb | In reply to @k900:0upti.me
You should now go look at the logs for that ok. Progress finally! See this:
Feb 07 22:01:03 netpleb-public-services named[524]: client @0x7f186e0f4768 127.0.0.1#40626/key rfc2136key.netpleb.com: signer "rfc2136key.netpleb.com" approved
Feb 07 22:01:03 netpleb-public-services named[524]: client @0x7f186e0f4768 127.0.0.1#40626/key rfc2136key.netpleb.com: updating zone 'netpleb.com/IN': deleting rrset at '_acme-challenge.jitsi.netpleb.com' TXT
Feb 07 22:01:03 netpleb-public-services named[524]: client @0x7f186e0f4768 127.0.0.1#40626/key rfc2136key.netpleb.com: updating zone 'netpleb.com/IN': adding an RR at '_acme-challenge.jitsi.netpleb.com' TXT "JMV6KVjVQtGlCFKSucMcbbCN8RqGY9_ZBZC3sVr9NW0"
Feb 07 22:01:03 netpleb-public-services named[524]: client @0x7f186e0f4768 127.0.0.1#40626/key rfc2136key.netpleb.com: updating zone 'netpleb.com/IN': error: journal open failed: unexpected error
Feb 07 22:01:03 netpleb-public-services named[524]: client @0x7f186c582368 127.0.0.1#50260/key rfc2136key.netpleb.com: signer "rfc2136key.netpleb.com" approved
| 22:13:24 |
| netpleb | it is unclear to me which journal it is talking about. I also did this setup verbatim from the nixos manual, but originally I did have NSD installed. Maybe that is related? | 22:20:02 |
| netpleb | * it is unclear to me which journal it is talking about? | 22:51:26 |
| netpleb | how do you guys install your zone? do you use something like environment.etc."bind/zones/example.com.zone".source = ./example.com.zone where ./example.com.zone is in the git repo (I am using flakes) | 23:29:36 |
| netpleb | * how do you guys install your zone? do you use something like environment.etc."bind/zones/example.com.zone".source = ./example.com.zone; where ./example.com.zone is in the git repo (I am using flakes) | 23:29:50 |
| 8 Feb 2024 |
| netpleb | finally fixed it...had to make a oneshot service that chmod --recursive named:named /etc/bind/zones so that named has permission to load some .jnl file which apparently it needs to do the acme stuff. | 01:09:16 |
|  symys joined the room. | 18:51:42 |
| netpleb | to round this "issue" out for anyone else who comes along trying to figure out something similar: it turns out that when you follow the "fully self-hosted example using bind" in the manual, there is a subtle thing which probably goes unnoticed for many, namely
bind.zones.*.file = "/var/db/bind/${name}"; is in a directory which needs to be read/writable by bind. On my machine that directory had not yet even been created, and never actually was. This is because I had instead set bind.zones.*.file = ./zone-file-in-my-git-repo which means that Nix put the zone file into the nix store (which is fine as far as I am concerned) but the problem is that bind tries to create the .jnl file right next to it when doing the acme updates, which it obviously cannot do. So that is what threw the permission denied error which took a while to track down. | 18:56:44 |
