Arian: I see.

Yes, my reason is that I need to migrate some older 23.05 NixOps deployments to newer NixOS.
For that to work well, I intended to start the old and new ones and compatibility-test them.
This does not work, because the AMIs are gone.
It betrays the entire point of fully declarative infrastructure-as-code that one cannot re-deploy the same thing at a later time.
Especially if it's only 1 year old.

I understand this isn't your or any NixOS contributor's fault, especially if AWS deletes public AMIs (no idea why they would do that, what is the point?).
You're referring to this, right?
https://aws.amazon.com/about-aws/whats-new/2022/03/amazon-machine-images-public-visibility-two-years/

But it also means that the way NixOps is written is completely unsuitable, and I'm not sure a lot of NixOps authors, contributors, or users are aware of this.
If public AMIs aren't a thing to last, NixOps should build an AMI itself an upload it.
(Probably people thought that they were a thing to last when that code was written.)
Indeed, no tool trying to provide reproducible deployments should be looking up AMIs from any list -- neither the old file amazon-ec2-amis.nix that you deleted, nor the new list at https://nixos.github.io/amis.

Completely independent question:

When I try to use NixOps to launch any newer AMI (it uses RunInstances), AWS errors with HTTP 400:

The parameter MetadataOptions is not recognized

I was puzzled by this, because I reproduced it curl where it also failed, and the request does not include the word MetadataOptions at all.
I just figured out that it seems to depend on the AMI launched.

AMIs for which this works/fails:

ami-0d6ee9d5e1c985df6   23.05.eu-central-1.x86_64-linux.hvm-ebs      WORKS
ami-0e7d1823ac80520e6   nixos/23.11.4976.79baff8812a0-x86_64-linux   ERRORS with `The parameter MetadataOptions is not recognized`
ami-0909c997a483185ec   nixos/23.11.7329.59a450646ec8-x86_64-linux   ERRORS with `The parameter MetadataOptions is not recognized`
ami-067999b24522c01c1   nixos/24.05.5809.4eb33fe664af-x86_64-linux   ERRORS with `The parameter MetadataOptions is not recognized`

Does anybody know what in the AMI could cause this?

In reply to @arianvp:matrix.org
I thought NixOps is unmaintained. I think they even removed the AWS plugin recently

Oh sure it is. Doesn't mean that all the startups using it to manage their infra evaporate though :D
With NixOps 2 never getting a real release and NixOps 4 already being the next "when it's done" software (I have some doubts that rewriting in Rust will solve community maintenance; it would certainly make debugging this stuff a lot harder than pdb allowed) ... NixOS 1.7 is the latest official stable release (TM) 🙃
Still works pretty well though.

I'm checking your replies now, very helpful, one moment

In reply to @arianvp:matrix.org
The only hit on google is this one stack overflow issue : https://stackoverflow.com/questions/76495875/cant-provision-ec2-instance-with-ansible

Yes, I saw that. My plan is to answer that StackOverflow question eventually.

maybe NixOps uses a very old version of the AWS SDK that is maybe incompatible with the AWS API?

The key thing is it can't' just be NixOps alone. Because NixOps (and the same curl command) launches the old instance but not the new. So it's not a plain AWS API removal.
I suspect it's something like "if you use an AMI that uses new feature X, you need to give some extra launch options".

This is the curl (credentials removed):

curl https://ec2.eu-west-1.amazonaws.com -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Authorization: AWS4-HMAC-SHA256 Credential=.../20241021/eu-west-1/ec2/aws4_request,SignedHeaders=host;x-amz-date,Signature=...' -H 'X-Amz-Date: 20241021T013814Z' --data 'Action=RunInstances&BlockDeviceMapping.1.DeviceName=%2Fdev%2Fxvda&BlockDeviceMapping.1.Ebs.DeleteOnTermination=true&BlockDeviceMapping.1.Ebs.Encrypted=false&BlockDeviceMapping.1.Ebs.SnapshotId=snap-0e6c56ddc0453c75a&BlockDeviceMapping.1.Ebs.VolumeSize=100&BlockDeviceMapping.1.Ebs.VolumeType=gp3&ClientToken=...&ImageId=ami-0e7d1823ac80520e6&InstanceInitiatedShutdownBehavior=stop&InstanceType=t2.medium&KeyName=my-nixops&MaxCount=1&MinCount=1&SecurityGroup.1=nixops-corp-test-2&UserData=...this only contains the SSH private key in base64...&Version=2014-10-01' -D -

Indeed Version=2014-10-01.
So what you're suggesting is quite likely.

In reply to @arianvp:matrix.org
The only hit on google is this one stack overflow issue : https://stackoverflow.com/questions/76495875/cant-provision-ec2-instance-with-ansible

Yes, I saw that. My plan is to answer that StackOverflow question eventually.

maybe NixOps uses a very old version of the AWS SDK that is maybe incompatible with the AWS API?

The key thing is it can't' just be NixOps alone. Because NixOps (and the same curl command) launches the old instance but not the new. So it's not a plain AWS API removal.
I suspect it's something like "if you use an AMI that uses new feature X, you need to give some extra launch options" (what you're suggesting, as I read along).

This is the curl (credentials removed):

curl https://ec2.eu-west-1.amazonaws.com -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Authorization: AWS4-HMAC-SHA256 Credential=.../20241021/eu-west-1/ec2/aws4_request,SignedHeaders=host;x-amz-date,Signature=...' -H 'X-Amz-Date: 20241021T013814Z' --data 'Action=RunInstances&BlockDeviceMapping.1.DeviceName=%2Fdev%2Fxvda&BlockDeviceMapping.1.Ebs.DeleteOnTermination=true&BlockDeviceMapping.1.Ebs.Encrypted=false&BlockDeviceMapping.1.Ebs.SnapshotId=snap-0e6c56ddc0453c75a&BlockDeviceMapping.1.Ebs.VolumeSize=100&BlockDeviceMapping.1.Ebs.VolumeType=gp3&ClientToken=...&ImageId=ami-0e7d1823ac80520e6&InstanceInitiatedShutdownBehavior=stop&InstanceType=t2.medium&KeyName=my-nixops&MaxCount=1&MinCount=1&SecurityGroup.1=nixops-corp-test-2&UserData=...this only contains the SSH private key in base64...&Version=2014-10-01' -D -

Indeed Version=2014-10-01.
So what you're suggesting is quite likely.

In reply to @nh2:matrix.org

Yes, I saw that. My plan is to answer that StackOverflow question eventually.

maybe NixOps uses a very old version of the AWS SDK that is maybe incompatible with the AWS API?

The key thing is it can't' just be NixOps alone. Because NixOps (and the same curl command) launches the old instance but not the new. So it's not a plain AWS API removal.
I suspect it's something like "if you use an AMI that uses new feature X, you need to give some extra launch options".

This is the curl (credentials removed):

curl https://ec2.eu-west-1.amazonaws.com -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Authorization: AWS4-HMAC-SHA256 Credential=.../20241021/eu-west-1/ec2/aws4_request,SignedHeaders=host;x-amz-date,Signature=...' -H 'X-Amz-Date: 20241021T013814Z' --data 'Action=RunInstances&BlockDeviceMapping.1.DeviceName=%2Fdev%2Fxvda&BlockDeviceMapping.1.Ebs.DeleteOnTermination=true&BlockDeviceMapping.1.Ebs.Encrypted=false&BlockDeviceMapping.1.Ebs.SnapshotId=snap-0e6c56ddc0453c75a&BlockDeviceMapping.1.Ebs.VolumeSize=100&BlockDeviceMapping.1.Ebs.VolumeType=gp3&ClientToken=...&ImageId=ami-0e7d1823ac80520e6&InstanceInitiatedShutdownBehavior=stop&InstanceType=t2.medium&KeyName=my-nixops&MaxCount=1&MinCount=1&SecurityGroup.1=nixops-corp-test-2&UserData=...this only contains the SSH private key in base64...&Version=2014-10-01' -D -

Indeed Version=2014-10-01.
So what you're suggesting is quite likely.