| 17 Jan 2025 |
 raitobezarius | turn on systemd debug logging and look for credential processing in the journal | 15:17:42 |
 phaer | Hm.. they work fine in initrd, but they don't get imported into stage2 for some reason.
Not importing credentials, $CREDENTIALS_DIRECTORY or $ENCRYPTED_CREDENTIALS_DIRECTORY already set.
No credentials passed from initrd.
That's a vm with tmpfs as /root etc.overlay and systemd in initrd both otherwise a pretty standard nixos-unstable.
| 15:48:07 |
 mlyx | Do you remount-root twice?
try copy /run/credentials/@system/KEY to /sysroot/run/credentials/ | 15:52:10 |
| phaer | I don't think so, and a grep in the log says I am only remounting root once.
copying the credentials manually should probably work but is more of a workaround imo. I am trying to find out why CREDENTIALS_DIRECTORY is already set here atm | 15:54:47 |
 @elvishjerricco:matrix.org | which thing is giving that message? Is it just PID1 or a service? | 16:09:03 |
| phaer | PID 1 as far as i can tell, i.e. no prefix in the log i get on the console https://gist.github.com/phaer/a992607df31fb18364264a77fa177e5f#file-log-txt-L5809 (sorry for the botched formatting) | 16:15:35 |
| @elvishjerricco:matrix.org | phaer: that's... interesting. Not usually what my journal looks like at all | 16:17:48 |
| @elvishjerricco:matrix.org | or did you copy the dmesg output or something? | 16:18:00 |
| phaer | I have the following kernel params set for debugging, if that's what you mean?
"console=hvc0"
"systemd.log_level=debug"
"systemd.log_target=console"
| 16:19:28 |
| phaer | and then the rest is just what i get from the serial console in my shell on the host, minus some terminal escape codes for colors | 16:20:23 |
| @elvishjerricco:matrix.org | Well you could have just run journalctl -b 0 to get a properly formatted log for the current boot I think | 16:20:23 |
| @elvishjerricco:matrix.org | the actual console output is formatted a little weird to make it visually appealing | 16:21:11 |
| @elvishjerricco:matrix.org | but it loses some of the stuff that gets logged in the journal | 16:21:21 |
| phaer | Thanks, that seems to confirm that the line about not importing the credentials because one of the env vars is already set is coming from pid 1 https://gist.github.com/phaer/b233e40012239e4411d56032867062ee#file-gistfile1-txt-L122 | 16:33:43 |
| @elvishjerricco:matrix.org | well that's very confusing. That log looks like something I don't understand is happening :P | 16:39:22 |
| @elvishjerricco:matrix.org | I'm fairly sure it's not even getting to switch-root, right? | 16:39:34 |
| @elvishjerricco:matrix.org | do you have any custom services in your stage 1 or 2? My paranoia makes me think that when I see logs I don't understand, it's probably an ordering cycle that caused systemd to blow up proper ordering almost entirely | 16:42:06 |
| @elvishjerricco:matrix.org | oh wait maybe I'm wrong. I didn't realize you had added a grep | 16:43:00 |
| phaer | That log is grepped for -i 'cred', if you'd like i can hapilly post the full one it's just a bit annoying to copy out of the serial, because i haven't set up sshd yet (injecting the pub keys without needing to rebuild images in the end is my goal here ;-P) | 16:43:07 |
| @elvishjerricco:matrix.org | so I wasn't seeing what I expected | 16:43:10 |
| @elvishjerricco:matrix.org | if it's got network access you can just pipe into nc termbin.com 9999 or something like that | 16:43:41 |
| phaer | ah right, that's a good tip thanks. Just read that after quickly rebuilding with sshd though, so heres the full thing https://gist.github.com/phaer/97bfce477b81c0247d79517fc7c1e2f2 | 16:50:24 |
| @elvishjerricco:matrix.org | phaer: That log makes it look like it never starts activation or switch-root | 16:56:08 |
| phaer | Ah right, I should see initrd-switch-root.service. Probably botched up the tmpfs root or something. will check out. Just weird that i end up in a state where systemctl status doesn't complain about anything and all my stage2 services are running 🤯 | 20:15:08 |
| @elvishjerricco:matrix.org | sounds like it did activate and switch-root, but for some reason it's just not in the journal? That doesn't make sense to me, but it's the only explanation I can think of | 20:38:08 |
| phaer | Thanks for the help so far, I'll take another look later this evening. If anyone here is curious enough, https://github.com/phaer/nixos-vm-on-macos/tree/cmdline-creds contains one nixosConfiguration. That's the one i am trying to boot here. With a custom system.build.vm attr. Maybe it's something in there 🤔 | 20:45:39 |
| @elvishjerricco:matrix.org | phaer: I wonder if the reason your journal seems cut short is just because journald fails to start in stage 2 or something. | 20:53:21 |
| @elvishjerricco:matrix.org | that seems extremely unlikely, but it would explain what we're seeing | 20:53:42 |
| @elvishjerricco:matrix.org | phaer: Can I run that repo on my M1 Pro MacBook? Like is it currently in a state where like nix run . will work or something? You've nerd sniped me on this :P | 20:58:31 |
| phaer | In reply to @elvishjerricco:matrix.org
phaer: Can I run that repo on my M1 Pro MacBook? Like is it currently in a state where like nix run . will work or something? You've nerd sniped me on this :P Haha, awesome 😅 yeah, the nix run command in the readme should start the VM. Only ran it on my M2 air so far, but it should even work on x86 macs | 21:00:21 |
