I have started to play with colmena just now. One thing that caught my eye is, to be able to override nix.extraOptions one needs to use lib.mkDefault, else both will get written to /etc/nix/nix.conf, e.g:

defaults = { ... }: {
  nix.extraOptions = lib.mkDefault ''
      min-free = ${toString (5 * 1024 * 1024 * 1024)}
  ''

How can I benchmark/profile what is causing colmena to be significantly slower than nixos-rebuild switch --flake? Two back-to-back executions below, no changes in between runs:

Colmena:

khalil:~/Documentos/NixOS % time colmena apply --on=andromeda
[INFO ] Using flake: git+file:///home/khalil/Documentos/NixOS
[INFO ] Enumerating nodes...
[INFO ] Selected 1 out of 4 hosts.
          ✅ 25s All done!
andromeda ✅ 19s Evaluated andromeda
andromeda ✅ 1s Built "/nix/store/yjscnkx6xy87s4z7x649dfz8b205j6nv-nixos-system-andromeda-25.
andromeda ✅ 1s Pushed system closure
andromeda ✅ 5s Activation successful                                                        
colmena apply --on=andromeda  10,13s user 5,66s system 42% cpu 37,122 total
khalil:~/Documentos/NixOS % time colmena apply --on=andromeda
[INFO ] Using flake: git+file:///home/khalil/Documentos/NixOS
[INFO ] Enumerating nodes...
[INFO ] Selected 1 out of 4 hosts.
          ✅ 25s All done!
andromeda ✅ 19s Evaluated andromeda
andromeda ✅ 1s Built "/nix/store/yjscnkx6xy87s4z7x649dfz8b205j6nv-nixos-system-andromeda-25.
andromeda ✅ 1s Pushed system closure
andromeda ✅ 5s Activation successful                                                        
colmena apply --on=andromeda  10,15s user 5,48s system 42% cpu 36,854 total
khalil:~/Documentos/NixOS % 

Nixos-rebuild switch --flake:

khalil:~/Documentos/NixOS % time nixos-rebuild switch --flake .#andromeda --target-host root@andromeda.host.ksantana.net
building the system configuration...
copying 0 paths...
Shared connection to andromeda.host.ksantana.net closed.
Shared connection to andromeda.host.ksantana.net closed.
stopping the following units: accounts-daemon.service, proc-sys-fs-binfmt_misc.automount, proc-sys-fs-binfmt_misc.mount, systemd-binfmt.service, systemd-tmpfiles-resetup.service
activating the configuration...
[agenix] creating new generation in /run/agenix.d/7
[agenix] decrypting secrets...
decrypting '/nix/store/maqzlgf99r1sf65xx2jiwzsmdm7jnzws-keepassxc.age' to '/run/agenix.d/7/keepassxc'...
decrypting '/nix/store/xlrhprsclp2l23mvm1bppsw6za819imn-restic-backup-edna.age' to '/run/agenix.d/7/restic-backup-edna'...
decrypting '/nix/store/ggr2z8ylnzl4wld7zb32nghn3qg9v82c-restic-backup-edna-remote.age' to '/run/agenix.d/7/restic-backup-edna-remote'...
decrypting '/nix/store/gp4clfixjmkhfvqmkcms6a4pb1jhxlc1-restic-backup-home-khalil.age' to '/run/agenix.d/7/restic-backup-home-khalil'...
decrypting '/nix/store/1wdan709p9lsx2jn9hzkf9sdapi20i1a-restic-backup-home-khalil-remote.age' to '/run/agenix.d/7/restic-backup-home-khalil-remote'...
decrypting '/nix/store/km4clf3936mji6b4h30yj5ccqcn1600x-tailscale.age' to '/run/agenix.d/7/tailscale'...
decrypting '/nix/store/2l8p93vz8jz4kc7z5339mrxjibpz46lj-users_khalil_passwd.age' to '/run/agenix.d/7/users_khalil_passwd'...
[agenix] symlinking new secrets to /run/agenix (generation 7)...
[agenix] removing old secrets (generation 6)...
[agenix] chowning...
setting up /etc...
reloading user units for khalil...
reloading user units for root...
restarting sysinit-reactivation.target
reloading the following units: dbus.service
restarting the following units: nix-daemon.service, polkit.service
starting the following units: accounts-daemon.service, systemd-tmpfiles-resetup.service
the following new units were started: systemd-localed.service
Shared connection to andromeda.host.ksantana.net closed.
Done. The new configuration is /nix/store/wzik0pvb0647gdnqfjlrp3j8dvm9mjh4-nixos-system-andromeda-25.11.20250527.4faa5f5
nixos-rebuild switch --flake .#andromeda --target-host   8,03s user 1,13s system 34% cpu 26,367 total
khalil:~/Documentos/NixOS % time nixos-rebuild switch --flake .#andromeda --target-host root@andromeda.host.ksantana.net
building the system configuration...
copying 0 paths...
Shared connection to andromeda.host.ksantana.net closed.
Shared connection to andromeda.host.ksantana.net closed.
activating the configuration...
[agenix] creating new generation in /run/agenix.d/8
[agenix] decrypting secrets...
decrypting '/nix/store/maqzlgf99r1sf65xx2jiwzsmdm7jnzws-keepassxc.age' to '/run/agenix.d/8/keepassxc'...
decrypting '/nix/store/xlrhprsclp2l23mvm1bppsw6za819imn-restic-backup-edna.age' to '/run/agenix.d/8/restic-backup-edna'...
decrypting '/nix/store/ggr2z8ylnzl4wld7zb32nghn3qg9v82c-restic-backup-edna-remote.age' to '/run/agenix.d/8/restic-backup-edna-remote'...
decrypting '/nix/store/gp4clfixjmkhfvqmkcms6a4pb1jhxlc1-restic-backup-home-khalil.age' to '/run/agenix.d/8/restic-backup-home-khalil'...
decrypting '/nix/store/1wdan709p9lsx2jn9hzkf9sdapi20i1a-restic-backup-home-khalil-remote.age' to '/run/agenix.d/8/restic-backup-home-khalil-remote'...
decrypting '/nix/store/km4clf3936mji6b4h30yj5ccqcn1600x-tailscale.age' to '/run/agenix.d/8/tailscale'...
decrypting '/nix/store/2l8p93vz8jz4kc7z5339mrxjibpz46lj-users_khalil_passwd.age' to '/run/agenix.d/8/users_khalil_passwd'...
[agenix] symlinking new secrets to /run/agenix (generation 8)...
[agenix] removing old secrets (generation 7)...
[agenix] chowning...
setting up /etc...
reloading user units for khalil...
reloading user units for root...
restarting sysinit-reactivation.target
Shared connection to andromeda.host.ksantana.net closed.
Done. The new configuration is /nix/store/wzik0pvb0647gdnqfjlrp3j8dvm9mjh4-nixos-system-andromeda-25.11.20250527.4faa5f5
nixos-rebuild switch --flake .#andromeda --target-host   0,12s user 0,07s system 3% cpu 4,908 total

My flake.nix:

{
  description = "KhalilSantana's NixOS configuration";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    nixpkgs-staging-next.url = "path:/mnt/data/@src-code/nixpkgs";
    pre-commit-hooks.url = "github:cachix/git-hooks.nix";
    home-manager.url = "github:nix-community/home-manager";
    home-manager.inputs.nixpkgs.follows = "nixpkgs";
    nix-flatpak.url = "github:gmodena/nix-flatpak"; # unstable branch. Use github:gmodena/nix-flatpak/?ref=<tag> to pin releases.
    agenix = {
      url = "github:ryantm/agenix";
      inputs.nixpkgs.follows = "nixpkgs"; # use the same nixpkgs as the main flake
      inputs.darwin.follows = ""; # optionally choose not to download darwin deps (saves some resources on Linux)
    };
  };

  outputs =
    {
      self,
      nixpkgs,
      nixpkgs-staging-next,
      nix-flatpak,
      home-manager,
      agenix,
      pre-commit-hooks,
      ...
    }:
    let
      supportedSystems = [
        "x86_64-linux"
        "aarch64-linux"
      ];

      forAllSystems = nixpkgs.lib.genAttrs supportedSystems;

      # Common function to create a host configuration
      mkHost =
        {
          hostName,
          system,
          extraModules ? [ ],
          extraSpecialArgs ? { },
        }:
        nixpkgs.lib.nixosSystem {
          inherit system;
          modules = [
            ./hosts/${hostName}/default.nix
            agenix.nixosModules.default
            nix-flatpak.nixosModules.nix-flatpak
            home-manager.nixosModules.home-manager
            {
              home-manager = {
                useGlobalPkgs = true;
                useUserPackages = true;
                users.khalil = ./home.nix;
              };
            }
          ] ++ extraModules;
          specialArgs = {
            stagingNextPkgs = import nixpkgs-staging-next { system = "aarch64-linux"; };
          } // extraSpecialArgs;
        };

      # Common function to create a colmena host
      mkColmenaHost =
        {
          hostName,
          system,
          extraModules ? [ ],
          extraSpecialArgs ? { },
        }:
        {
          imports = [
            ./hosts/${hostName}/default.nix
            agenix.nixosModules.default
            nix-flatpak.nixosModules.nix-flatpak
            home-manager.nixosModules.home-manager
            {
              home-manager = {
                useGlobalPkgs = true;
                useUserPackages = true;
                users.khalil = ./home.nix;
              };
            }
          ] ++ extraModules;
          nixpkgs.system = system;
          deployment.targetHost = "${hostName}.host.ksantana.net";
          _module.args = {
            stagingNextPkgs = import nixpkgs-staging-next { system = "aarch64-linux"; };
          } // extraSpecialArgs;
        };

    in
    {
      formatter = forAllSystems (system: nixpkgs.legacyPackages.${system}.nixfmt-rfc-style);
      # https://github.com/cachix/git-hooks.nix
      checks = forAllSystems (
        system:
        let
          pkgs = import nixpkgs { inherit system; };
        in
        {
          pre-commit-check = pre-commit-hooks.lib.${system}.run {
            src = ./.;
            hooks = {
              nixfmt-rfc-style.enable = true;
              statix.enable = true;
              flake-checker.enable = true;
              deadnix.enable = true;
              gitleaks = {
                enable = true;
                name = "Gitleaks secrets scan";
                entry = "${pkgs.gitleaks}/bin/gitleaks git --staged --baseline-path gitleaks-report.json -v --no-color --no-banner";
                language = "system";
                pass_filenames = false;
                stages = [ "pre-commit" ];
              };
            };
          };
        }
      );

      devShells = forAllSystems (system: {
        default = nixpkgs.legacyPackages.${system}.mkShell {
          inherit (self.checks.${system}.pre-commit-check) shellHook;
          buildInputs = self.checks.${system}.pre-commit-check.enabledPackages;
        };
      });

      colmena = {
        meta = {
          nixpkgs = import nixpkgs {
            system = "x86_64-linux";
            overlays = [ ];
          };
          nodeSpecialArgs = {
            stagingNextPkgs = import nixpkgs-staging-next {
              system = "aarch64-linux";
            };
          };
        };

        "andromeda" = mkColmenaHost {
          hostName = "andromeda";
          system = "x86_64-linux";
          extraModules = [ { boot.binfmt.emulatedSystems = [ "aarch64-linux" ]; } ];
        };

        "umbrella" = mkColmenaHost {
          hostName = "umbrella";
          system = "x86_64-linux";
          extraModules = [ { boot.binfmt.emulatedSystems = [ "aarch64-linux" ]; } ];
        };

        "ocloud-ks" = mkColmenaHost {
          hostName = "ocloud-ks";
          system = "aarch64-linux";
        };

        "littlesombrero" = mkColmenaHost {
          hostName = "littlesombrero";
          system = "aarch64-linux";
        };
      };

      nixosConfigurations = {
        andromeda = mkHost {
          hostName = "andromeda";
          system = "x86_64-linux";
        };
        umbrella = mkHost {
          hostName = "umbrella";
          system = "x86_64-linux";
        };
        ocloud-ks = mkHost {
          hostName = "ocloud-ks";
          system = "aarch64-linux";
        };
        littlesombrero = mkHost {
          hostName = "littlesombrero";
          system = "aarch64-linux";
        };
      };
    };
}

(I don't feel too confortable sharing the full git repo, but I can probably produce a shallow version of it if required, with minimal sensitive info)

That seems to get nixos-rebuild --flake performance closer to colmena's, but there's still a significant difference (if I'm reading the output of zshs time correctly, that is).

colmena apply --on=andromeda --nix-option eval-cache false  9,84s user 5,36s system 41% cpu 36,341 total
nixos-rebuild switch --flake .#andromeda --target-host  --option eval-cache   7,88s user 1,11s system 58% cpu 15,299 total

I was trying to setup profiling in nix using this[1] to figure out why colmena is slower, but didn't quite succeed (I think I got a -ENOSPC or something due to /tmp/ as tmpfs or something). Do you think something like this would be useful to debug this performance behaviour?

[1] - https://github.com/crabdancing/nix-flamegraph

In reply to @khalil:ksantana.net

I was trying to setup profiling in nix using this[1] to figure out why colmena is slower, but didn't quite succeed (I think I got a -ENOSPC or something due to /tmp/ as tmpfs or something). Do you think something like this would be useful to debug this performance behaviour?

[1] - https://github.com/crabdancing/nix-flamegraph

Heh, it seems that plotting the flamegraph it is not quite possible in my system with 32GB of RAM.:

khalil:~/Documentos/NixOS % nix run github:crabdancing/nix-flamegraph -- --target  .#colmenaHive.toplevel.andromeda
Started `nix eval` against target...
Done. Running `stack-collapse`...
Done. Running `inferno-flamegraph`...
Error: Io(Kind(OutOfMemory))
nix run github:crabdancing/nix-flamegraph -- --target   655,34s user 224,26s system 97% cpu 14:59,05 total

[ 3404.188809] __vm_enough_memory: pid: 43928, comm: inferno-flamegr, bytes: 104871428096 not enough memory for the allocation
[ 3404.188815] __vm_enough_memory: pid: 43928, comm: inferno-flamegr, bytes: 104871497728 not enough memory for the allocation
[ 3404.188817] __vm_enough_memory: pid: 43928, comm: inferno-flamegr, bytes: 104871563264 not enough memory for the allocation

It's been long overdue, but Colmena 0.4.0 is finally here: