Hey all, I'm just starting to get into cross compiling (x68 -> aarch64 for rpi 4b), and have some questions about how to best approach this. I'll start with my understanding and situation to make sure I didn't make a mistake:

• I'm using flakes, which don't exactly have first class support for cross compilation.
• I'm manually creating my pi's nixosConfigurations key and making my own nixpkgs.pkgs for it.
• It seems like for this pkgs argument, I have two options: The first, is create it with localSystem == crossSystem == "aarch64-linux". If I do this, I'd need to rely on binmft style emulation to build from my x86 machine. The second, is create it with localSystem = x86-linux; crossSystem = "aarch64-linux". If I do this, I'm doing real cross compilation.
• With the former I can use cache.nixos.org to grab the aarch64 components, but I'd need to emulate-build everything else.
• With the latter I can't use the cache at all, so I need to build everything.

Ideally I could use the cache and cross compile only if needed, but it looks like I can't do that without manually choosing specific packages to be cross compiled with pkgsCross attributes in the first method, right? And if I did want this, then my buildSystem would need to support emulation, right?

My buildSystem is currently Ubuntu, so I can't just flip the nixos switch for binfmt emulation. I could probably apply the binfmt config manually if I want to go the emulation route, right?

I suppose I might just be frustrated that I can't default crossSystem and opportunistically grab the non-cross packages from the aarch64 cache, though I know the technical reason why (store paths are functions of all inputs, and crossSystem is an input). I'm guessing this is currently an unsolved issue?

Thanks!

With the latter I can't use the cache at all, so I need to build everything.

There's some packages cached https://github.com/NixOS/nixpkgs/blob/75c025c6135ca219cded2592343bef3daa4b39b9/pkgs/top-level/release-cross.nix#L239
https://github.com/NixOS/nixpkgs/blob/75c025c6135ca219cded2592343bef3daa4b39b9/pkgs/top-level/release-cross.nix#L78-L105

My buildSystem is currently Ubuntu, so I can't just flip the nixos switch for binfmt emulation. I could probably apply the binfmt config manually if I want to go the emulation route, right?

Yup binfmt is possible to setup on all distros

Ideally I could use the cache and cross compile only if needed, but it looks like I can't do that without manually choosing specific packages to be cross compiled with pkgsCross attributes in the first method, right? And if I did want this, then my buildSystem would need to support emulation, right?

Can't automatically pick the packages to get from the cache and which to cross-compile. You can however make a second native nixpkgs instance to pick packages from

Ideally I could use the cache and cross compile only if needed, but it looks like I can't do that without manually choosing specific packages to be cross compiled with pkgsCross attributes in the first method, right? And if I did want this, then my buildSystem would need to support emulation, right?

Can't automatically pick the packages to get from the cache and which to cross-compile. You can however make a second native nixpkgs instance to pick packages from, you could even overlay them.

Can't automatically pick the packages to get from the cache and which to cross-compile.

Could I introspect the cache somehow (outside of nix, ahead of time) and add a nixpkgs overlay based on pname to select between ( native | cross ), hoping that the hashes will then match?

You can however make a second native nixpkgs instance to pick packages from, you could even overlay them.

If I do set up binfmt emulation, and I'm using a stable nixos branch that hydra has cached, and not building custom packages, should I expect a 99% cache hit rate, and then the last bit is essentially making the sdcard images which I'd need to do locally?

nix-forecast

Very nice, looks like something that will be very useful for a/b comparisons. Thanks for the pointer!

hydra-check

Good to know, but probably more useful for those actively hacking on nixpkgs