| 18 May 2024 |
matthewcroughan | The way I see it there are only two problems remaining with disko:
- no compression for diskImages output
- no hostPlatform recognition (build on hostPlatform vm and run binfmt inside of that accelerated kvm machine)
| 00:09:50 |
matthewcroughan | 2 is a bit of an architectural issue that I looked into earlier, the way disko and even vmTools are constructed, doesn't easily allow you to fix that | 00:10:31 |
matthewcroughan | atm vmTools will fall back to tcg, and basically hang, never completing the build | 00:11:20 |
matthewcroughan | instead, vmTools should execute a VM for the hostplatform, with binfmt enabled. But vmTools is so minimal that putting binfmt in there wouldn't re-use the nixpkgs machinery (boot.binfmt.emulatedSystems) | 00:12:05 |
matthewcroughan | * instead, vmTools should execute a VM for the hostPlatform, with binfmt enabled, and run all the scripts for arm64, But vmTools is so minimal that putting binfmt in there wouldn't re-use the nixpkgs machinery (boot.binfmt.emulatedSystems) | 00:12:22 |
matthewcroughan | The use case is building a raspberry pi image, of course :D | 00:12:34 |
matthewcroughan | x86 kvm -> nested arm64 binfmt, is much much much faster than arm64 -> arm64 , which is what currently happens | 00:15:07 |
matthewcroughan | * The way I see it there are only two problems remaining with disko:
- no compression for diskImages output
- no hostPlatform recognition (build on hostPlatform vm and run binfmt for arm64 inside of that accelerated kvm machine)
| 00:15:32 |
matthewcroughan | PRs welcome, I know.. but it's really quite difficult to re-imagine the make-disk-image function and vmTools with hostPlatform conditions | 00:16:28 |
matthewcroughan | * x86 kvm -> nested arm64 binfmt, is much much much faster than arm64 binfmt -> arm64 tcg vm , which is what currently happens if you try to build an arm64 disk image on x86 | 00:17:12 |
matthewcroughan | * x86 kvm -> nested arm64 binfmt, is much much much faster than arm64 binfmt -> arm64 tcg vm , which is what currently happens if you try to build an arm64 disk image on an x86 host | 00:17:23 |
matthewcroughan | corpo-disko-images> + findmnt /dev/disk/by-partlabel/disk-disk1-firmware /mnt/firmware
corpo-disko-images> + mount /dev/disk/by-partlabel/disk-disk1-firmware /mnt/firmware -t vfat -o defaults -o X-mount.mkdir
corpo-disko-images> + /nix/store/gc39rhvazg28s7y80c4c0a24x3hsv92p-postMountHook.sh
corpo-disko-images> + rm -rf /tmp/tmp.GkPSSeFxnb
corpo-disko-images> warning: the group 'nixbld' specified in 'build-users-group' does not exist
corpo-disko-images> cp: cannot access '/nix/store/fikh9lqcr2vfz3ncq3sihqzhjw482yps-mbrola-voices-3.3/data/gr1': Cannot allocate memory
corpo-disko-images> cp: cannot access '/nix/store/nw17gpncwy29l0r8kaxzss6nb4c1bh6s-linux-firmware-20240513-zstd/lib/firmware/qcom/sm8250': Cannot allocate memory
corpo-disko-images> [ 122.745756] reboot: Power down
| 01:02:03 |
matthewcroughan | I ran into this Cannot allocate memory issue again, but it happens non-deterministically. It only happens every now and then.. very strange | 01:02:29 |
lassulus | AFAIK there was a way to compress the images disko generates with a hook | 08:16:01 |
lassulus | https://github.com/nix-community/disko/blob/master/lib/make-disk-image.nix#L6 | 08:16:34 |
| 19 May 2024 |
matthewcroughan | In reply to @lassulus:lassul.us https://github.com/nix-community/disko/blob/master/lib/make-disk-image.nix#L6 Oh yes, but this is more manual and not a module option, which I think it should be | 11:27:02 |
matthewcroughan | I may figure that out and contribute it today | 11:27:27 |
matthewcroughan | I have another issue though, which is that I need the bootloader to be on a partition other than the root partition | 11:27:55 |
matthewcroughan | This is pretty close to working
{ pkgs, ... }:
let
configTxt = pkgs.writeText "config.txt" ''
[pi4]
kernel=u-boot-rpi4.bin
enable_gic=1
# Otherwise the resolution will be weird in most cases, compared to
# what the pi3 firmware does by default.
disable_overscan=1
# Supported in newer board revisions
arm_boost=1
[cm4]
# Enable host mode on the 2711 built-in XHCI USB controller.
# This line should be removed if the legacy DWC2 controller is required
# (e.g. for USB device mode) or if USB support is not required.
otg_mode=1
[all]
# Boot in 64-bit mode.
arm_64bit=1
# U-Boot needs this to work, regardless of whether UART is actually used or not.
# Look in arch/arm/mach-bcm283x/Kconfig in the U-Boot tree to see if this is still
# a requirement in the future.
enable_uart=1
# Prevent the firmware from smashing the framebuffer setup done by the mainline kernel
# when attempting to show low-voltage or overtemperature warnings.
avoid_warnings=1
'';
in
{
disko.devices = {
disk = {
disk1 = {
imageSize = "10G";
type = "disk";
device = "/dev/mmcblk0";
postCreateHook = ''
lsblk
sgdisk -A 2:set:2 /dev/vdb
'';
content = {
type = "gpt";
partitions = {
firmware = {
size = "30M";
type = "EF00";
content = {
type = "filesystem";
format = "vfat";
mountpoint = "/firmware";
postMountHook = toString (pkgs.writeScript "postMountHook.sh" ''
(cd ${pkgs.raspberrypifw}/share/raspberrypi/boot && cp bootcode.bin fixup*.dat start*.elf *.dtb /mnt/firmware/)
cp ${pkgs.ubootRaspberryPi4_64bit}/u-boot.bin /mnt/firmware/u-boot-rpi4.bin
cp ${configTxt} /mnt/firmware/config.txt
'');
};
};
root = {
name = "root";
size = "100%";
content = {
type = "filesystem";
format = "bcachefs";
mountpoint = "/";
};
};
};
};
};
};
};
}
| 11:29:10 |
matthewcroughan | * This is pretty close to working
{ pkgs, ... }:
let
configTxt = pkgs.writeText "config.txt" ''
[pi4]
kernel=u-boot-rpi4.bin
enable_gic=1
# Otherwise the resolution will be weird in most cases, compared to
# what the pi3 firmware does by default.
disable_overscan=1
# Supported in newer board revisions
arm_boost=1
[cm4]
# Enable host mode on the 2711 built-in XHCI USB controller.
# This line should be removed if the legacy DWC2 controller is required
# (e.g. for USB device mode) or if USB support is not required.
otg_mode=1
[all]
# Boot in 64-bit mode.
arm_64bit=1
# U-Boot needs this to work, regardless of whether UART is actually used or not.
# Look in arch/arm/mach-bcm283x/Kconfig in the U-Boot tree to see if this is still
# a requirement in the future.
enable_uart=1
# Prevent the firmware from smashing the framebuffer setup done by the mainline kernel
# when attempting to show low-voltage or overtemperature warnings.
avoid_warnings=1
'';
in
{
disko.devices = {
disk = {
disk1 = {
imageSize = "10G";
type = "disk";
device = "/dev/mmcblk0";
postCreateHook = ''
lsblk
sgdisk -A 2:set:2 /dev/vdb
'';
content = {
type = "gpt";
partitions = {
firmware = {
size = "30M";
type = "EF00";
content = {
type = "filesystem";
format = "vfat";
mountpoint = "/firmware";
postMountHook = toString (pkgs.writeScript "postMountHook.sh" ''
(cd ${pkgs.raspberrypifw}/share/raspberrypi/boot && cp bootcode.bin fixup*.dat start*.elf *.dtb /mnt/firmware/)
cp ${pkgs.ubootRaspberryPi4_64bit}/u-boot.bin /mnt/firmware/u-boot-rpi4.bin
cp ${configTxt} /mnt/firmware/config.txt
'');
};
};
root = {
name = "root";
size = "100%";
content = {
type = "filesystem";
format = "bcachefs";
mountpoint = "/";
};
};
};
};
};
};
};
}
| 11:29:15 |
matthewcroughan | Note sgdisk -A 2:set:2 /dev/vdb because of Pi/U-Boot interactions when using GPT only without a PMBR | 11:29:41 |
matthewcroughan | U-boot requires the legacy bios bootable flag on the root partition where /boot/extlinux resides | 11:30:00 |
matthewcroughan | but u-boot doesn't support loading extlinux from bcachefs yet, so I need a little fat/ext4 partition containing it | 11:30:46 |
matthewcroughan | systemd-repart can't do bcachefs due to needing loopback though | 11:33:04 |
matthewcroughan | Oh, in addition to compression not being a module option, there's also image resizing, which would be a really nice to have, though maybe it's not in disko's domain, I am not sure about it | 11:37:37 |
lassulus | I think compression is fine as a hook. But we need more documentation for that | 11:37:54 |
matthewcroughan | Disko already "pollutes?" the NixOS Configuration with the automatic filesystems options, so maybe it should also handle disk resizing if enabled as a module option | 11:38:39 |
matthewcroughan | the same way systemd-repart does | 11:38:43 |
matthewcroughan | Arguably, hardware-configuration.nix and associated stuff is the real pollution, and disko/repart fix that | 11:39:20 |
matthewcroughan | Found https://github.com/nix-community/disko/pull/384 which sorta documents the compression you can apply with extraPostVM | 11:41:52 |