Completely independent question:

When I try to use NixOps to launch any newer AMI (it uses RunInstances), AWS errors with HTTP 400:

The parameter MetadataOptions is not recognized

I was puzzled by this, because I reproduced it curl where it also failed, and the request does not include the word MetadataOptions at all.
I just figured out that it seems to depend on the AMI launched.

AMIs for which this works/fails:

ami-0d6ee9d5e1c985df6   23.05.eu-central-1.x86_64-linux.hvm-ebs      WORKS
ami-0e7d1823ac80520e6   nixos/23.11.4976.79baff8812a0-x86_64-linux   ERRORS with `The parameter MetadataOptions is not recognized`
ami-0909c997a483185ec   nixos/23.11.7329.59a450646ec8-x86_64-linux   ERRORS with `The parameter MetadataOptions is not recognized`
ami-067999b24522c01c1   nixos/24.05.5809.4eb33fe664af-x86_64-linux   ERRORS with `The parameter MetadataOptions is not recognized`

Does anybody know what in the AMI could cause this?

In reply to @arianvp:matrix.org
I thought NixOps is unmaintained. I think they even removed the AWS plugin recently

Oh sure it is. Doesn't mean that all the startups using it to manage their infra evaporate though :D
With NixOps 2 never getting a real release and NixOps 4 already being the next "when it's done" software (I have some doubts that rewriting in Rust will solve community maintenance; it would certainly make debugging this stuff a lot harder than pdb allowed) ... NixOS 1.7 is the latest official stable release (TM) 🙃
Still works pretty well though.

I'm checking your replies now, very helpful, one moment

In reply to @arianvp:matrix.org
The only hit on google is this one stack overflow issue : https://stackoverflow.com/questions/76495875/cant-provision-ec2-instance-with-ansible

Yes, I saw that. My plan is to answer that StackOverflow question eventually.

maybe NixOps uses a very old version of the AWS SDK that is maybe incompatible with the AWS API?

The key thing is it can't' just be NixOps alone. Because NixOps (and the same curl command) launches the old instance but not the new. So it's not a plain AWS API removal.
I suspect it's something like "if you use an AMI that uses new feature X, you need to give some extra launch options".

This is the curl (credentials removed):

curl https://ec2.eu-west-1.amazonaws.com -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Authorization: AWS4-HMAC-SHA256 Credential=.../20241021/eu-west-1/ec2/aws4_request,SignedHeaders=host;x-amz-date,Signature=...' -H 'X-Amz-Date: 20241021T013814Z' --data 'Action=RunInstances&BlockDeviceMapping.1.DeviceName=%2Fdev%2Fxvda&BlockDeviceMapping.1.Ebs.DeleteOnTermination=true&BlockDeviceMapping.1.Ebs.Encrypted=false&BlockDeviceMapping.1.Ebs.SnapshotId=snap-0e6c56ddc0453c75a&BlockDeviceMapping.1.Ebs.VolumeSize=100&BlockDeviceMapping.1.Ebs.VolumeType=gp3&ClientToken=...&ImageId=ami-0e7d1823ac80520e6&InstanceInitiatedShutdownBehavior=stop&InstanceType=t2.medium&KeyName=my-nixops&MaxCount=1&MinCount=1&SecurityGroup.1=nixops-corp-test-2&UserData=...this only contains the SSH private key in base64...&Version=2014-10-01' -D -

Indeed Version=2014-10-01.
So what you're suggesting is quite likely.

In reply to @arianvp:matrix.org
The only hit on google is this one stack overflow issue : https://stackoverflow.com/questions/76495875/cant-provision-ec2-instance-with-ansible

Yes, I saw that. My plan is to answer that StackOverflow question eventually.

maybe NixOps uses a very old version of the AWS SDK that is maybe incompatible with the AWS API?

The key thing is it can't' just be NixOps alone. Because NixOps (and the same curl command) launches the old instance but not the new. So it's not a plain AWS API removal.
I suspect it's something like "if you use an AMI that uses new feature X, you need to give some extra launch options" (what you're suggesting, as I read along).

This is the curl (credentials removed):

curl https://ec2.eu-west-1.amazonaws.com -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Authorization: AWS4-HMAC-SHA256 Credential=.../20241021/eu-west-1/ec2/aws4_request,SignedHeaders=host;x-amz-date,Signature=...' -H 'X-Amz-Date: 20241021T013814Z' --data 'Action=RunInstances&BlockDeviceMapping.1.DeviceName=%2Fdev%2Fxvda&BlockDeviceMapping.1.Ebs.DeleteOnTermination=true&BlockDeviceMapping.1.Ebs.Encrypted=false&BlockDeviceMapping.1.Ebs.SnapshotId=snap-0e6c56ddc0453c75a&BlockDeviceMapping.1.Ebs.VolumeSize=100&BlockDeviceMapping.1.Ebs.VolumeType=gp3&ClientToken=...&ImageId=ami-0e7d1823ac80520e6&InstanceInitiatedShutdownBehavior=stop&InstanceType=t2.medium&KeyName=my-nixops&MaxCount=1&MinCount=1&SecurityGroup.1=nixops-corp-test-2&UserData=...this only contains the SSH private key in base64...&Version=2014-10-01' -D -

Indeed Version=2014-10-01.
So what you're suggesting is quite likely.

In reply to @nh2:matrix.org

Yes, I saw that. My plan is to answer that StackOverflow question eventually.

maybe NixOps uses a very old version of the AWS SDK that is maybe incompatible with the AWS API?

The key thing is it can't' just be NixOps alone. Because NixOps (and the same curl command) launches the old instance but not the new. So it's not a plain AWS API removal.
I suspect it's something like "if you use an AMI that uses new feature X, you need to give some extra launch options".

This is the curl (credentials removed):

curl https://ec2.eu-west-1.amazonaws.com -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Authorization: AWS4-HMAC-SHA256 Credential=.../20241021/eu-west-1/ec2/aws4_request,SignedHeaders=host;x-amz-date,Signature=...' -H 'X-Amz-Date: 20241021T013814Z' --data 'Action=RunInstances&BlockDeviceMapping.1.DeviceName=%2Fdev%2Fxvda&BlockDeviceMapping.1.Ebs.DeleteOnTermination=true&BlockDeviceMapping.1.Ebs.Encrypted=false&BlockDeviceMapping.1.Ebs.SnapshotId=snap-0e6c56ddc0453c75a&BlockDeviceMapping.1.Ebs.VolumeSize=100&BlockDeviceMapping.1.Ebs.VolumeType=gp3&ClientToken=...&ImageId=ami-0e7d1823ac80520e6&InstanceInitiatedShutdownBehavior=stop&InstanceType=t2.medium&KeyName=my-nixops&MaxCount=1&MinCount=1&SecurityGroup.1=nixops-corp-test-2&UserData=...this only contains the SSH private key in base64...&Version=2014-10-01' -D -

Indeed Version=2014-10-01.
So what you're suggesting is quite likely.