In reply to @colemickens:matrix.org
With sops-nix having support for cloud KVM, and a scheme I've found for supporting multiple user zfs keys... I think some somewhat interesting things could be built that would be highly appealing to certain types of orgs.

In reply to @colemickens:matrix.org
I think maybe this is OT and maybe I should throw up a repo even just so we can spitball ideas, but I am thinking along the same lines. In many cases, disk layout will be known ahead of time and I'm thinking a heuristic for "matching" to a selected disk layout to a booted machine, it notices the disks are not zpool members and kicks in.

In reply to @colemickens:matrix.org
phaer: finally circling back to look at this and I noticed that your zfs sets "encryption=off". why is that?

In reply to @colemickens:matrix.org
I was all into my idea of iPXE booting Oracle VMs into some sort of installer like that, but I'm not sure it gets me a whole lot over your kexec approach, since kexec seems to work even on OCI aarch64 VMs.

bobvanderlinden_: I looked at your branch today and while I like what I see, there's one or two things you should consider. First off, a huge PR which touches most of nixos/ is unlikely to find happy mergers and likely to find happy reverters if it breaks something (which is of course more likely with huge PRs). I saw you split out the systemd splitting stuff and I think that's a good way to go forward.

The second thing is that I noticed you just replace stage-1.nix. This is of course the way to go in the future, but imo it would be easier to adopt if it was opt-in at first, opt-out in a later release and no option anymore in an even later release. The reason for that is that I find it hard to believe that all use cases of all users are working when the PR is merged (there are just so many of them), so being able to go back to the old behavior could help. For simple systems, curious users (like myself) could adopt the new stage-1 in 22.05 (or whenever the PR is merged) already while people with more complex setups can wait until their stuff works. A thing you could do here (just my thoughts) would be to provide a secondary interface, maybe something like boot.initramfs (Wikipedia says we have a initramfs, not an initrd) with visibility = "shallow" in the manual so people can discover the option. This way you could cut all ties with the old initrd interface and start from a clean slate. While most options still need counterparts in the new initrd, we don't necessarily have to keep all types and implications of these options

Janne Heß: Thanks for looking into it! I completely agree. I'd like to do https://github.com/NixOS/nixpkgs/pull/164016 in stages and the individual commits are the smallest I could make the steps. A next stage is also ready, but I'd like to send that as a separate PR. That one makes the options units, services, slices, etc reusable, so they can be reused in systemd, systemd.user as well as initrd.systemd. I'd like to use the same units/services/etc structure, so that we have something to build on further.

I totally agree that stage-1.nix should be made compatible with an option to switch between the two. I worked on a local branch myself in the past where that was the case, but since this PR (https://github.com/NixOS/nixpkgs/pull/120015) seemed to be progress further along I thought I'd resurrect that one. To move on that one further I'd like to be able to add systemd services to initrd properly, hence why I started with the refactoring of systemd. Once I can add services, I'd like to add a preDeviceCommands, postDeviceCommands, postMountCommands, etc services that will essentially emulate the current initrd boot process.

boot.initrd.systemd.enable = true; sounds like a viable option.

I know big PRs are quite annoying to review and merge. If I can make the changes smaller, let me know. I can also submit the individual commits as separate PRs if that's desired. For some of the changes I can even make sure that the build hashes of the tests do not change (an actual 'no functionality has changed' refactor). As long as I know the path I'm walking is one that won't get stalled. Let me know what I can do to move things forward :+1:.

Btw, I really appreciate your message here, I was quite glad to see it. I'll install the Element mobile/desktop to keep up with notifications here.

Janne Heß: Thanks for looking into it! I completely agree. I'd like to do https://github.com/NixOS/nixpkgs/pull/164016 in stages and the individual commits are the smallest I could make the steps. A next stage is also ready, but I'd like to send that as a separate PR. That one makes the options units, services, slices, etc reusable, so they can be reused in systemd, systemd.user as well as initrd.systemd. I'd like to use the same units/services/etc structure, so that we have something to build on further.

I totally agree that stage-1.nix should be made compatible with an option to switch between the two. I worked on a local branch myself in the past where that was the case, but since this PR (https://github.com/NixOS/nixpkgs/pull/120015) seemed to be progress further along I thought I'd resurrect that one. To move on that one further I'd like to be able to add systemd services to initrd properly, hence why I started with the refactoring of systemd. Once I can add services, I'd like to add a preDeviceCommands, postDeviceCommands, postMountCommands, etc services that will essentially emulate the current initrd boot process.

boot.initrd.systemd.enable = true; sounds like a viable option.

I know big PRs are quite annoying to review and merge. If I can make the changes smaller, let me know. I can also submit the individual commits as separate PRs if that's desired. For some of the changes I can even make sure that the build hashes of the tests do not change (an actual 'no functionality has changed' refactor). As long as I know the path I'm walking is one that won't get stalled (and result in neverending rebases and conflict resolution). Let me know what I can do to move things forward :+1:.

Btw, I really appreciate your message here, I was quite glad to see it. I'll install the Element mobile/desktop to keep up with notifications here.

Janne Heß: Thanks for looking into it! I completely agree. I'd like to do https://github.com/NixOS/nixpkgs/pull/164016 in stages and the individual commits are the smallest I could make the steps. A next stage is also ready, but I'd like to send that as a separate PR. That one makes the options units, services, slices, etc reusable, so they can be reused in systemd, systemd.user as well as initrd.systemd. I'd like to use the same units/services/etc structure, so that we have something to build on further.

I totally agree that stage-1.nix should be made compatible with an option to switch between the two. I worked on a local branch myself in the past where that was the case, but since this PR (https://github.com/NixOS/nixpkgs/pull/120015) seemed to be progress further along I thought I'd resurrect that one. To move on that one further I'd like to be able to add systemd services to initrd properly, hence why I started with the refactoring of systemd. Once I can add services, I'd like to add a preDeviceCommands, postDeviceCommands, postMountCommands, etc services that will essentially emulate the current initrd boot process.

boot.initrd.systemd.enable = true; or boot.initramfs.enable = true are viable options. I still need to dive deeper into this than I currently have.

I know big PRs are quite annoying to review and merge. If I can make the changes smaller, let me know. I can also submit the individual commits as separate PRs if that's desired. For some of the changes I can even make sure that the build hashes of the tests do not change (an actual 'no functionality has changed' refactor). As long as I know the path I'm walking is one that won't get stalled (and result in neverending rebases and conflict resolution). Let me know what I can do to move things forward :+1:.

Btw, I really appreciate your message here, I was quite glad to see it. I'll install the Element mobile/desktop to keep up with notifications here.

Janne Heß: Thanks for looking into it! I completely agree. I'd like to do https://github.com/NixOS/nixpkgs/pull/164016 in stages and the individual commits are the smallest I could make the steps. A next stage is also ready, but I'd like to send that as a separate PR. That one makes the options units, services, slices, etc reusable, so they can be reused in systemd, systemd.user as well as initrd.systemd. I'd like to use the same units/services/etc structure, so that we have something to build on further.

I totally agree that stage-1.nix should be made compatible with an option to switch between the two. I worked on a local branch myself in the past where that was the case, but since this PR (https://github.com/NixOS/nixpkgs/pull/120015) seemed to be progressed further along I thought I'd resurrect that one. To move on that one further I'd like to be able to add systemd services to initrd properly, hence why I started with the refactoring of systemd. Once I can add services, I'd like to add a preDeviceCommands, postDeviceCommands, postMountCommands, etc services that will essentially emulate the current initrd boot process.

boot.initrd.systemd.enable = true; or boot.initramfs.enable = true are viable options. I still need to dive deeper into this than I currently have.

I know big PRs are quite annoying to review and merge. If I can make the changes smaller, let me know. I can also submit the individual commits as separate PRs if that's desired. For some of the changes I can even make sure that the build hashes of the tests do not change (an actual 'no functionality has changed' refactor). As long as I know the path I'm walking is one that won't get stalled (and result in neverending rebases and conflict resolution). Let me know what I can do to move things forward :+1:.

Btw, I really appreciate your message here, I was quite glad to see it. I'll install the Element mobile/desktop to keep up with notifications here.