In reply to @piegames:matrix.org
So you have three services connecting to postgres, and one of them fails during upgrade. That's not an implementation detail, that's a fundamental problem IMO

migrations as such need ot be kind of managed as involved service groups that may or may not share required rollbacks

yes but this is a metadata problem somewhat, right?

for full safety you need service deployment granularity that enables migrations in steps where both older and newer versons currently avaliable can work with the data in question

if your app supports it, yes; if not, you don't need it, as long as you always know how to "rollback an app code" and "rollback an app data", this is the only thing you need and this can always be done if you know where all the state is and you have Nix semantics

maintenance modes for the applications that can happen independ of the code deployment are going to be important as well

yes, maintenance mode is one of the low hanging fruit for those procedures, nixops had a PR which enabled many niceties on this, we discussed it again in nixos deployments but I don't think it went somewhere

so starting points like "having restorable backups" are definitively nice to have but applications themselfes also need some extras

at least, they would prevent people losing all their bookmarks in nextcloud or losing all their docs in hedgedocs, because some contributors blindly believe that automatic rollback is fine because everyone is responsible and do backups (which is not totally true)

something fun and unique one could do with a system like nixos, is to have non gc roots of avaliable application versions that link to the state versions discoverable (like have a version of the old app, the new app and the migrate command, and the systemd service will pick the executable based on current state ) - classically this would be done with tagged container images

this can already be achieved by manipulating nix profiles properly I believe