With the latter I can't use the cache at all, so I need to build everything.

There's some packages cached https://github.com/NixOS/nixpkgs/blob/75c025c6135ca219cded2592343bef3daa4b39b9/pkgs/top-level/release-cross.nix#L239
https://github.com/NixOS/nixpkgs/blob/75c025c6135ca219cded2592343bef3daa4b39b9/pkgs/top-level/release-cross.nix#L78-L105

My buildSystem is currently Ubuntu, so I can't just flip the nixos switch for binfmt emulation. I could probably apply the binfmt config manually if I want to go the emulation route, right?

Yup binfmt is possible to setup on all distros

Ideally I could use the cache and cross compile only if needed, but it looks like I can't do that without manually choosing specific packages to be cross compiled with pkgsCross attributes in the first method, right? And if I did want this, then my buildSystem would need to support emulation, right?

Can't automatically pick the packages to get from the cache and which to cross-compile. You can however make a second native nixpkgs instance to pick packages from

Ideally I could use the cache and cross compile only if needed, but it looks like I can't do that without manually choosing specific packages to be cross compiled with pkgsCross attributes in the first method, right? And if I did want this, then my buildSystem would need to support emulation, right?

Can't automatically pick the packages to get from the cache and which to cross-compile. You can however make a second native nixpkgs instance to pick packages from, you could even overlay them.

Can't automatically pick the packages to get from the cache and which to cross-compile.

Could I introspect the cache somehow (outside of nix, ahead of time) and add a nixpkgs overlay based on pname to select between ( native | cross ), hoping that the hashes will then match?

You can however make a second native nixpkgs instance to pick packages from, you could even overlay them.

If I do set up binfmt emulation, and I'm using a stable nixos branch that hydra has cached, and not building custom packages, should I expect a 99% cache hit rate, and then the last bit is essentially making the sdcard images which I'd need to do locally?

nix-forecast

Very nice, looks like something that will be very useful for a/b comparisons. Thanks for the pointer!

hydra-check

Good to know, but probably more useful for those actively hacking on nixpkgs

I cross compile to my raspis

What configuration settings do you use here? I'm new to this one too, and I see I can choose (mainline | proprietary) kernel, (nixpkgs stable | nix-hardware | raspberry-pi-nix) for hardware support.

  boot = {
    supportedFilesystems = {
      btrfs = lib.mkForce false;
      cifs = lib.mkForce false;
      f2fs = lib.mkForce false;
      ntfs = lib.mkForce false;
      xfs = lib.mkForce false;
      zfs = lib.mkForce false;
    };
    # Make the kernel much lighter to build, otherwise needs 30G+ ram
    kernelPatches = [
      {
        name = "disable-bpf";
        patch = null;
        extraConfig = ''
          DEBUG_INFO_BTF n
          CONFIG_DEBUG_INFO_BTF n
        '';
      }
    ];
    initrd.allowMissingModules = true;

    tmp.useTmpfs = true;
  };

Didn't change the kernel

obstacle to making it all automatic with nix-forecast

I figured the biggest issue would be that it's not easy to map cached store paths back to attr / package names

Also got these to not compile gtk

  nixpkgs.overlays = [
    (final: prev: {
      stoken = prev.stoken.override { withGTK3 = false; };
      alsa-utils = prev.alsa-utils.override {
        withPipewireLib = false;
        fftw = null;
      };
      alsa-plugins = prev.alsa-plugins.override {
        libpulseaudio = null;
        speexdsp = null;
      };
      sdl2 = prev.sdl2.override {
        pipewireSupport = false;
        pulseaudioSupport = false;
        x11Support = false;
        jackSupport = false;
      };
      ffmpeg = prev.ffmpeg_7-headless.override {
        withSpeex = false;
        withOpenmpt = false;
      };
    })
  ];

Also got these to not compile gtk and some other packages

  nixpkgs.overlays = [
    (final: prev: {
      stoken = prev.stoken.override { withGTK3 = false; };
      alsa-utils = prev.alsa-utils.override {
        withPipewireLib = false;
        fftw = null;
      };
      alsa-plugins = prev.alsa-plugins.override {
        libpulseaudio = null;
        speexdsp = null;
      };
      sdl2 = prev.sdl2.override {
        pipewireSupport = false;
        pulseaudioSupport = false;
        x11Support = false;
        jackSupport = false;
      };
      ffmpeg = prev.ffmpeg_7-headless.override {
        withSpeex = false;
        withOpenmpt = false;
      };
    })
  ];