Now I have two other problems.

1. For some reason the DHCP server built into networkd that I use on my pi router requires my networkd clients to have dhcpV4Config.ClientIdentifier = "mac"; if I want static IPs to work. So now I have to configure that in initrd.
2. For some reason systemd-timesyncd.service is failing in stage 2 now with Failed to allocate manager: Permission denied

In reply to @mlyx:matrix.org
Has anyone tested boot.initrd.systemd.enable = true in kexec initrd? Seems that it doesn't work. I don't know how to debug initrd. What's the reason? https://github.com/nix-community/nixos-images

In reply to @phaer:matrix.org
Not exactly what you are asking for but https://github.com/dep-sys/nix-dabei uses boot.initrd.systemd.enable (and the networkd PR) and kexecs into an initrd (without stage-2) with enough tools, you might at least find some inspiration there?
W.r.t. debugging: one could increase log verbosity (see kernelParams in the repo), run a shell on console (e.g. https://github.com/dep-sys/nix-dabei/blob/main/modules/auto-installer.nix#L61) or just start sshd if networkd is included in your initrd :)

In reply to @me:linj.tech
great project! May I ask what is the trick to reduce the size of initrd compared to netboot?

In reply to @elvishjerricco:matrix.org
mlyx: You would make a new initrd systemd service that has wantedBy = ["initrd.target"];. You also want the service to be ordered after file systems are mounted, but if you don't have DefaultDependencies=no, then it'll have After=local-fs.target by default anyway. But eventually we're going to change it so systemd-initrd uses initrd-fs.target like we're supposed to, which services don't get ordered after by default

  boot.initrd.systemd.services = {
    impermanence-root = {
      wantedBy = [ "initrd.target" ];
      after = [ "local-fs.target" ];
      serviceConfig = {
        Type = "oneshot";
        ExecStart = "${pkgs.zfs}/bin/zfs rollback -r zroot/local/root@blank";
      };
    };
  };