boot.initrd.luks.devices = {
    # TODO what if one disk is missing
    "nixos-1" = {
      allowDiscards = true;
      bypassWorkqueues = true;
      device = "/dev/disk/by-label/luks-1";
      # fallbackToPassword = true;
      # TODO x-initrd.attach
      # TODO x-systemd.device-timeout=
      crypttabExtraOpts = [ "nofail" "x-systemd.device-timeout=10" ];
    };
    "nixos-2" = {
      allowDiscards = true;
      bypassWorkqueues = true;
      device = "/dev/disk/by-label/luks-2";
      # fallbackToPassword = true;
      crypttabExtraOpts = [ "nofail" "x-systemd.device-timeout=10" ];
    };
  };

  fileSystems."/" =
    { device = "none";
      fsType = "tmpfs";
      options = [ "defaults" "size=500M" "mode=755" ];
    };

  # TODO what if one disk is missing
  fileSystems."/boot" =
    { device = "/dev/disk/by-label/efi-1";
      fsType = "vfat";
      options = [ "nofail" ];
    };

  # TODO what if one disk is missing
  fileSystems."/boot2" =
    { device = "/dev/disk/by-label/efi-2";
      fsType = "vfat";
      options = [ "nofail" ];
    };

  # TODO what if one disk is missing
  fileSystems."/nix" =
    { # device = "/dev/disk/by-label/nixos";
      device = "/dev/disk/by-uuid/f93cfbf1-e9b4-46ca-b7cb-6f3fb1554fbb";
      fsType = "btrfs";
      options = [ "degraded" "noatime" "ssd" "compress=zstd:1" "space_cache=v2" "subvol=@nix" ];
    };

  fileSystems."/var" =
    { # device = "/dev/disk/by-label/nixos";
      device = "/dev/disk/by-uuid/f93cfbf1-e9b4-46ca-b7cb-6f3fb1554fbb";
      fsType = "btrfs";
      options = [ "degraded" "noatime" "ssd" "compress=zstd:1" "space_cache=v2" "subvol=@var" ];
    };

  fileSystems."/home" =
    { # device = "/dev/disk/by-label/nixos";
      device = "/dev/disk/by-uuid/f93cfbf1-e9b4-46ca-b7cb-6f3fb1554fbb";
      fsType = "btrfs";
      options = [ "degraded" "noatime" "ssd" "compress=zstd:1" "space_cache=v2" "subvol=@home" ];
      neededForBoot = true;
    };

  fileSystems."/state" =
    { # device = "/dev/disk/by-label/nixos";
      device = "/dev/disk/by-uuid/f93cfbf1-e9b4-46ca-b7cb-6f3fb1554fbb";
      fsType = "btrfs";
      options = [ "degraded" "noatime" "ssd" "compress=zstd:1" "space_cache=v2" "subvol=@state" ];
      neededForBoot = true;
    };

In reply to @elvishjerricco:matrix.org
gotcha, so it's normal to just pick one of the devices in a btrfs raid array as the device?

In reply to @elvishjerricco:matrix.org
once you're in the emergency shell in the systemd stage 1, you can probably manually mount the degraded btrfs root and let it continue with systemctl default

linj: to clarify a bit, you want x-initrd.{attach,mount} in the /etc/crypttab and /etc/fstab of stage 2. Their function is to tell systemd two things.

1. In stage 1, when the root fs is mounted at /sysroot, check /sysroot/etc/{crypttab,fstab} for entries with these options by re-running generators and generating stage 1 units for them with /sysroot prefixed for FSes. This is why we have the initrd-root-fs.target and initrd-fs.target sync points; they represent /sysroot being mounted and x-initrd.mount FSes being mounted.

2. They tell the shutdown process not to mess with these things. The shutdown ramfs (if enabled) can attempt to safely unmount and detach them, but the regular shutdown process shouldn't.

So this gets complicated for NixOS. We want these things written in our stage 2 crypttab and fstab files so that shutdown respects them, but the problem is that we can't rely on them existing for stage 1. For instance if you have root on tmpfs, then /sysroot/etc doesn't even exist, let alone contain the necessary crypttab and fstab entries with x-initrd.* options. And even for people whose root is persistent, those files need to be updated by initrd-nixos-activation.service before we can rely on them, which itself already requires all the stage 1 FSes to be mounted.

ElvishJerricco: After reading your repl, systemd doc and nixpkgs pr, I have several questions:

1. Do we need /sysroot/etc/{crypttab,fstab}? I think currently we have put FSes needed in /etc/fstab and /etc/crypttab of initrd.
2. Is this the shutdown ramfs you mentioned above? Does that pr handles unmounting "root on btrfs raid1 on luks on two disks" correctly? Some FSes is missing in journalctl -b -1 -g Unmounting. Don't know if that is related though.

ElvishJerricco: After reading your reply, systemd doc and nixpkgs pr, I have several questions:

1. Do we need /sysroot/etc/{crypttab,fstab}? I think currently we have put FSes needed in /etc/fstab and /etc/crypttab of initrd.
2. Is this the shutdown ramfs you mentioned above? Does that pr handles unmounting "root on btrfs raid1 on luks on two disks" correctly? Some FSes is missing in journalctl -b -1 -g Unmounting. Don't know if that is related though.

ElvishJerricco: After reading your reply, systemd doc and nixpkgs pr, I have several questions:

1. Do we need /sysroot/etc/{crypttab,fstab}? I think currently we have put FSes needed in /etc/fstab and /etc/crypttab of initrd.
2. Is this the shutdown ramfs you mentioned above? Does that pr handle unmounting "root on btrfs raid1 on luks on two disks" correctly? Some FSes is missing in journalctl -b -1 -g Unmounting. Don't know if that is related though.