Secrets should be encrypted to several keys:

• the ssh host public key of each system that needs it, to be decrypted at boot / activation
• the user public key of each admin that needs to edit or change the config, such as when re-encrypting to add a new host

Note, in particular, that neither of these happens during build (but, yes, switch involves activation that should not involve a user key)