How can I benchmark/profile what is causing colmena to be significantly slower than nixos-rebuild switch --flake? Two back-to-back executions below, no changes in between runs:

Colmena:

khalil:~/Documentos/NixOS % time colmena apply --on=andromeda
[INFO ] Using flake: git+file:///home/khalil/Documentos/NixOS
[INFO ] Enumerating nodes...
[INFO ] Selected 1 out of 4 hosts.
          ✅ 25s All done!
andromeda ✅ 19s Evaluated andromeda
andromeda ✅ 1s Built "/nix/store/yjscnkx6xy87s4z7x649dfz8b205j6nv-nixos-system-andromeda-25.
andromeda ✅ 1s Pushed system closure
andromeda ✅ 5s Activation successful                                                        
colmena apply --on=andromeda  10,13s user 5,66s system 42% cpu 37,122 total
khalil:~/Documentos/NixOS % time colmena apply --on=andromeda
[INFO ] Using flake: git+file:///home/khalil/Documentos/NixOS
[INFO ] Enumerating nodes...
[INFO ] Selected 1 out of 4 hosts.
          ✅ 25s All done!
andromeda ✅ 19s Evaluated andromeda
andromeda ✅ 1s Built "/nix/store/yjscnkx6xy87s4z7x649dfz8b205j6nv-nixos-system-andromeda-25.
andromeda ✅ 1s Pushed system closure
andromeda ✅ 5s Activation successful                                                        
colmena apply --on=andromeda  10,15s user 5,48s system 42% cpu 36,854 total
khalil:~/Documentos/NixOS % 

Nixos-rebuild switch --flake:

khalil:~/Documentos/NixOS % time nixos-rebuild switch --flake .#andromeda --target-host root@andromeda.host.ksantana.net
building the system configuration...
copying 0 paths...
Shared connection to andromeda.host.ksantana.net closed.
Shared connection to andromeda.host.ksantana.net closed.
stopping the following units: accounts-daemon.service, proc-sys-fs-binfmt_misc.automount, proc-sys-fs-binfmt_misc.mount, systemd-binfmt.service, systemd-tmpfiles-resetup.service
activating the configuration...
[agenix] creating new generation in /run/agenix.d/7
[agenix] decrypting secrets...
decrypting '/nix/store/maqzlgf99r1sf65xx2jiwzsmdm7jnzws-keepassxc.age' to '/run/agenix.d/7/keepassxc'...
decrypting '/nix/store/xlrhprsclp2l23mvm1bppsw6za819imn-restic-backup-edna.age' to '/run/agenix.d/7/restic-backup-edna'...
decrypting '/nix/store/ggr2z8ylnzl4wld7zb32nghn3qg9v82c-restic-backup-edna-remote.age' to '/run/agenix.d/7/restic-backup-edna-remote'...
decrypting '/nix/store/gp4clfixjmkhfvqmkcms6a4pb1jhxlc1-restic-backup-home-khalil.age' to '/run/agenix.d/7/restic-backup-home-khalil'...
decrypting '/nix/store/1wdan709p9lsx2jn9hzkf9sdapi20i1a-restic-backup-home-khalil-remote.age' to '/run/agenix.d/7/restic-backup-home-khalil-remote'...
decrypting '/nix/store/km4clf3936mji6b4h30yj5ccqcn1600x-tailscale.age' to '/run/agenix.d/7/tailscale'...
decrypting '/nix/store/2l8p93vz8jz4kc7z5339mrxjibpz46lj-users_khalil_passwd.age' to '/run/agenix.d/7/users_khalil_passwd'...
[agenix] symlinking new secrets to /run/agenix (generation 7)...
[agenix] removing old secrets (generation 6)...
[agenix] chowning...
setting up /etc...
reloading user units for khalil...
reloading user units for root...
restarting sysinit-reactivation.target
reloading the following units: dbus.service
restarting the following units: nix-daemon.service, polkit.service
starting the following units: accounts-daemon.service, systemd-tmpfiles-resetup.service
the following new units were started: systemd-localed.service
Shared connection to andromeda.host.ksantana.net closed.
Done. The new configuration is /nix/store/wzik0pvb0647gdnqfjlrp3j8dvm9mjh4-nixos-system-andromeda-25.11.20250527.4faa5f5
nixos-rebuild switch --flake .#andromeda --target-host   8,03s user 1,13s system 34% cpu 26,367 total
khalil:~/Documentos/NixOS % time nixos-rebuild switch --flake .#andromeda --target-host root@andromeda.host.ksantana.net
building the system configuration...
copying 0 paths...
Shared connection to andromeda.host.ksantana.net closed.
Shared connection to andromeda.host.ksantana.net closed.
activating the configuration...
[agenix] creating new generation in /run/agenix.d/8
[agenix] decrypting secrets...
decrypting '/nix/store/maqzlgf99r1sf65xx2jiwzsmdm7jnzws-keepassxc.age' to '/run/agenix.d/8/keepassxc'...
decrypting '/nix/store/xlrhprsclp2l23mvm1bppsw6za819imn-restic-backup-edna.age' to '/run/agenix.d/8/restic-backup-edna'...
decrypting '/nix/store/ggr2z8ylnzl4wld7zb32nghn3qg9v82c-restic-backup-edna-remote.age' to '/run/agenix.d/8/restic-backup-edna-remote'...
decrypting '/nix/store/gp4clfixjmkhfvqmkcms6a4pb1jhxlc1-restic-backup-home-khalil.age' to '/run/agenix.d/8/restic-backup-home-khalil'...
decrypting '/nix/store/1wdan709p9lsx2jn9hzkf9sdapi20i1a-restic-backup-home-khalil-remote.age' to '/run/agenix.d/8/restic-backup-home-khalil-remote'...
decrypting '/nix/store/km4clf3936mji6b4h30yj5ccqcn1600x-tailscale.age' to '/run/agenix.d/8/tailscale'...
decrypting '/nix/store/2l8p93vz8jz4kc7z5339mrxjibpz46lj-users_khalil_passwd.age' to '/run/agenix.d/8/users_khalil_passwd'...
[agenix] symlinking new secrets to /run/agenix (generation 8)...
[agenix] removing old secrets (generation 7)...
[agenix] chowning...
setting up /etc...
reloading user units for khalil...
reloading user units for root...
restarting sysinit-reactivation.target
Shared connection to andromeda.host.ksantana.net closed.
Done. The new configuration is /nix/store/wzik0pvb0647gdnqfjlrp3j8dvm9mjh4-nixos-system-andromeda-25.11.20250527.4faa5f5
nixos-rebuild switch --flake .#andromeda --target-host   0,12s user 0,07s system 3% cpu 4,908 total

My flake.nix:

{
  description = "KhalilSantana's NixOS configuration";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    nixpkgs-staging-next.url = "path:/mnt/data/@src-code/nixpkgs";
    pre-commit-hooks.url = "github:cachix/git-hooks.nix";
    home-manager.url = "github:nix-community/home-manager";
    home-manager.inputs.nixpkgs.follows = "nixpkgs";
    nix-flatpak.url = "github:gmodena/nix-flatpak"; # unstable branch. Use github:gmodena/nix-flatpak/?ref=<tag> to pin releases.
    agenix = {
      url = "github:ryantm/agenix";
      inputs.nixpkgs.follows = "nixpkgs"; # use the same nixpkgs as the main flake
      inputs.darwin.follows = ""; # optionally choose not to download darwin deps (saves some resources on Linux)
    };
  };

  outputs =
    {
      self,
      nixpkgs,
      nixpkgs-staging-next,
      nix-flatpak,
      home-manager,
      agenix,
      pre-commit-hooks,
      ...
    }:
    let
      supportedSystems = [
        "x86_64-linux"
        "aarch64-linux"
      ];

      forAllSystems = nixpkgs.lib.genAttrs supportedSystems;

      # Common function to create a host configuration
      mkHost =
        {
          hostName,
          system,
          extraModules ? [ ],
          extraSpecialArgs ? { },
        }:
        nixpkgs.lib.nixosSystem {
          inherit system;
          modules = [
            ./hosts/${hostName}/default.nix
            agenix.nixosModules.default
            nix-flatpak.nixosModules.nix-flatpak
            home-manager.nixosModules.home-manager
            {
              home-manager = {
                useGlobalPkgs = true;
                useUserPackages = true;
                users.khalil = ./home.nix;
              };
            }
          ] ++ extraModules;
          specialArgs = {
            stagingNextPkgs = import nixpkgs-staging-next { system = "aarch64-linux"; };
          } // extraSpecialArgs;
        };

      # Common function to create a colmena host
      mkColmenaHost =
        {
          hostName,
          system,
          extraModules ? [ ],
          extraSpecialArgs ? { },
        }:
        {
          imports = [
            ./hosts/${hostName}/default.nix
            agenix.nixosModules.default
            nix-flatpak.nixosModules.nix-flatpak
            home-manager.nixosModules.home-manager
            {
              home-manager = {
                useGlobalPkgs = true;
                useUserPackages = true;
                users.khalil = ./home.nix;
              };
            }
          ] ++ extraModules;
          nixpkgs.system = system;
          deployment.targetHost = "${hostName}.host.ksantana.net";
          _module.args = {
            stagingNextPkgs = import nixpkgs-staging-next { system = "aarch64-linux"; };
          } // extraSpecialArgs;
        };

    in
    {
      formatter = forAllSystems (system: nixpkgs.legacyPackages.${system}.nixfmt-rfc-style);
      # https://github.com/cachix/git-hooks.nix
      checks = forAllSystems (
        system:
        let
          pkgs = import nixpkgs { inherit system; };
        in
        {
          pre-commit-check = pre-commit-hooks.lib.${system}.run {
            src = ./.;
            hooks = {
              nixfmt-rfc-style.enable = true;
              statix.enable = true;
              flake-checker.enable = true;
              deadnix.enable = true;
              gitleaks = {
                enable = true;
                name = "Gitleaks secrets scan";
                entry = "${pkgs.gitleaks}/bin/gitleaks git --staged --baseline-path gitleaks-report.json -v --no-color --no-banner";
                language = "system";
                pass_filenames = false;
                stages = [ "pre-commit" ];
              };
            };
          };
        }
      );

      devShells = forAllSystems (system: {
        default = nixpkgs.legacyPackages.${system}.mkShell {
          inherit (self.checks.${system}.pre-commit-check) shellHook;
          buildInputs = self.checks.${system}.pre-commit-check.enabledPackages;
        };
      });

      colmena = {
        meta = {
          nixpkgs = import nixpkgs {
            system = "x86_64-linux";
            overlays = [ ];
          };
          nodeSpecialArgs = {
            stagingNextPkgs = import nixpkgs-staging-next {
              system = "aarch64-linux";
            };
          };
        };

        "andromeda" = mkColmenaHost {
          hostName = "andromeda";
          system = "x86_64-linux";
          extraModules = [ { boot.binfmt.emulatedSystems = [ "aarch64-linux" ]; } ];
        };

        "umbrella" = mkColmenaHost {
          hostName = "umbrella";
          system = "x86_64-linux";
          extraModules = [ { boot.binfmt.emulatedSystems = [ "aarch64-linux" ]; } ];
        };

        "ocloud-ks" = mkColmenaHost {
          hostName = "ocloud-ks";
          system = "aarch64-linux";
        };

        "littlesombrero" = mkColmenaHost {
          hostName = "littlesombrero";
          system = "aarch64-linux";
        };
      };

      nixosConfigurations = {
        andromeda = mkHost {
          hostName = "andromeda";
          system = "x86_64-linux";
        };
        umbrella = mkHost {
          hostName = "umbrella";
          system = "x86_64-linux";
        };
        ocloud-ks = mkHost {
          hostName = "ocloud-ks";
          system = "aarch64-linux";
        };
        littlesombrero = mkHost {
          hostName = "littlesombrero";
          system = "aarch64-linux";
        };
      };
    };
}

(I don't feel too confortable sharing the full git repo, but I can probably produce a shallow version of it if required, with minimal sensitive info)

That seems to get nixos-rebuild --flake performance closer to colmena's, but there's still a significant difference (if I'm reading the output of zshs time correctly, that is).

colmena apply --on=andromeda --nix-option eval-cache false  9,84s user 5,36s system 41% cpu 36,341 total
nixos-rebuild switch --flake .#andromeda --target-host  --option eval-cache   7,88s user 1,11s system 58% cpu 15,299 total

I was trying to setup profiling in nix using this[1] to figure out why colmena is slower, but didn't quite succeed (I think I got a -ENOSPC or something due to /tmp/ as tmpfs or something). Do you think something like this would be useful to debug this performance behaviour?

[1] - https://github.com/crabdancing/nix-flamegraph

In reply to @khalil:ksantana.net

I was trying to setup profiling in nix using this[1] to figure out why colmena is slower, but didn't quite succeed (I think I got a -ENOSPC or something due to /tmp/ as tmpfs or something). Do you think something like this would be useful to debug this performance behaviour?

[1] - https://github.com/crabdancing/nix-flamegraph

Heh, it seems that plotting the flamegraph it is not quite possible in my system with 32GB of RAM.:

khalil:~/Documentos/NixOS % nix run github:crabdancing/nix-flamegraph -- --target  .#colmenaHive.toplevel.andromeda
Started `nix eval` against target...
Done. Running `stack-collapse`...
Done. Running `inferno-flamegraph`...
Error: Io(Kind(OutOfMemory))
nix run github:crabdancing/nix-flamegraph -- --target   655,34s user 224,26s system 97% cpu 14:59,05 total

[ 3404.188809] __vm_enough_memory: pid: 43928, comm: inferno-flamegr, bytes: 104871428096 not enough memory for the allocation
[ 3404.188815] __vm_enough_memory: pid: 43928, comm: inferno-flamegr, bytes: 104871497728 not enough memory for the allocation
[ 3404.188817] __vm_enough_memory: pid: 43928, comm: inferno-flamegr, bytes: 104871563264 not enough memory for the allocation

It's been long overdue, but Colmena 0.4.0 is finally here:

In reply to @freexploit:trafkin.com
I just want to say thank you 😄 for now I've been using colmena for more than a year to manage my home lab and it has been great