In reply to @infinidoge:inx.moe
It home manager it should let you set all normal settings

Well, I guess I dont' understand the syntax for this:

Host *
    IdentityAgent ~/.1password/agent.sock

host github.com
    hostname ssh.github.com
    port 443
    user damccull
    identityfile ~/.ssh/id_ed25519.pub

How do I build this with the options? I could definitely use home.file, but I think the permissions would then be wrong.

Something like

programs.ssh.matchBlock."*".identityAgent = "`~/.1passwordagent.sock";
programs.ssh.matchBlock."github.com" {
  hostname = "ssh.github.com"
  # ... more options?
};

Something like this?

programs.ssh.matchBlock."*".identityAgent = "`~/.1passwordagent.sock";
programs.ssh.matchBlock."github.com" {
  hostname = "ssh.github.com"
  # ... more options?
};