| 22 Oct 2024 |
Arian | If updating nixops is not an option I'm afraid you'll have to upload a custom AMI with that setting unset :( | 00:57:09 |
Arian | But that'll get you into trouble as well as the ec2 nixos modules rely on IMDSv2 these days | 00:59:38 |
Arian | Wait nah that's fine | 01:00:52 |
Arian | IMDSv2 works on IMDSv1 images. | 01:01:02 |
Arian | Just not the other way around | 01:01:06 |
nh2 | In reply to @arianvp:matrix.org The only hit on google is this one stack overflow issue : https://stackoverflow.com/questions/76495875/cant-provision-ec2-instance-with-ansible Yes, I saw that. My plan is to answer that StackOverflow question eventually.
maybe NixOps uses a very old version of the AWS SDK that is maybe incompatible with the AWS API?
The key thing is it can't' just be NixOps alone. Because NixOps (and the same curl command) launches the old instance but not the new. So it's not a plain AWS API removal. I suspect it's something like "if you use an AMI that uses new feature X, you need to give some extra launch options".
This is the curl (credentials removed):
curl https://ec2.eu-west-1.amazonaws.com -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Authorization: AWS4-HMAC-SHA256 Credential=.../20241021/eu-west-1/ec2/aws4_request,SignedHeaders=host;x-amz-date,Signature=...' -H 'X-Amz-Date: 20241021T013814Z' --data 'Action=RunInstances&BlockDeviceMapping.1.DeviceName=%2Fdev%2Fxvda&BlockDeviceMapping.1.Ebs.DeleteOnTermination=true&BlockDeviceMapping.1.Ebs.Encrypted=false&BlockDeviceMapping.1.Ebs.SnapshotId=snap-0e6c56ddc0453c75a&BlockDeviceMapping.1.Ebs.VolumeSize=100&BlockDeviceMapping.1.Ebs.VolumeType=gp3&ClientToken=...&ImageId=ami-0e7d1823ac80520e6&InstanceInitiatedShutdownBehavior=stop&InstanceType=t2.medium&KeyName=my-nixops&MaxCount=1&MinCount=1&SecurityGroup.1=nixops-corp-test-2&UserData=...this only contains the SSH private key in base64...&Version=2014-10-01' -D -
Indeed Version=2014-10-01. So what you're suggesting is quite likely.
| 01:01:15 |
nh2 | In reply to @arianvp:matrix.org The only hit on google is this one stack overflow issue : https://stackoverflow.com/questions/76495875/cant-provision-ec2-instance-with-ansible * Yes, I saw that. My plan is to answer that StackOverflow question eventually.
maybe NixOps uses a very old version of the AWS SDK that is maybe incompatible with the AWS API?
The key thing is it can't' just be NixOps alone. Because NixOps (and the same curl command) launches the old instance but not the new. So it's not a plain AWS API removal. I suspect it's something like "if you use an AMI that uses new feature X, you need to give some extra launch options" (what you're suggesting, as I read along).
This is the curl (credentials removed):
curl https://ec2.eu-west-1.amazonaws.com -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Authorization: AWS4-HMAC-SHA256 Credential=.../20241021/eu-west-1/ec2/aws4_request,SignedHeaders=host;x-amz-date,Signature=...' -H 'X-Amz-Date: 20241021T013814Z' --data 'Action=RunInstances&BlockDeviceMapping.1.DeviceName=%2Fdev%2Fxvda&BlockDeviceMapping.1.Ebs.DeleteOnTermination=true&BlockDeviceMapping.1.Ebs.Encrypted=false&BlockDeviceMapping.1.Ebs.SnapshotId=snap-0e6c56ddc0453c75a&BlockDeviceMapping.1.Ebs.VolumeSize=100&BlockDeviceMapping.1.Ebs.VolumeType=gp3&ClientToken=...&ImageId=ami-0e7d1823ac80520e6&InstanceInitiatedShutdownBehavior=stop&InstanceType=t2.medium&KeyName=my-nixops&MaxCount=1&MinCount=1&SecurityGroup.1=nixops-corp-test-2&UserData=...this only contains the SSH private key in base64...&Version=2014-10-01' -D -
Indeed Version=2014-10-01. So what you're suggesting is quite likely.
| 01:01:54 |
nh2 | So I think you pinpointed it exactly | 01:02:06 |
Arian | In reply to @nh2:matrix.org
Yes, I saw that. My plan is to answer that StackOverflow question eventually.
maybe NixOps uses a very old version of the AWS SDK that is maybe incompatible with the AWS API?
The key thing is it can't' just be NixOps alone. Because NixOps (and the same curl command) launches the old instance but not the new. So it's not a plain AWS API removal. I suspect it's something like "if you use an AMI that uses new feature X, you need to give some extra launch options".
This is the curl (credentials removed):
curl https://ec2.eu-west-1.amazonaws.com -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Authorization: AWS4-HMAC-SHA256 Credential=.../20241021/eu-west-1/ec2/aws4_request,SignedHeaders=host;x-amz-date,Signature=...' -H 'X-Amz-Date: 20241021T013814Z' --data 'Action=RunInstances&BlockDeviceMapping.1.DeviceName=%2Fdev%2Fxvda&BlockDeviceMapping.1.Ebs.DeleteOnTermination=true&BlockDeviceMapping.1.Ebs.Encrypted=false&BlockDeviceMapping.1.Ebs.SnapshotId=snap-0e6c56ddc0453c75a&BlockDeviceMapping.1.Ebs.VolumeSize=100&BlockDeviceMapping.1.Ebs.VolumeType=gp3&ClientToken=...&ImageId=ami-0e7d1823ac80520e6&InstanceInitiatedShutdownBehavior=stop&InstanceType=t2.medium&KeyName=my-nixops&MaxCount=1&MinCount=1&SecurityGroup.1=nixops-corp-test-2&UserData=...this only contains the SSH private key in base64...&Version=2014-10-01' -D -
Indeed Version=2014-10-01. So what you're suggesting is quite likely. That Imdssupport:V2 Option on the AMI causes an implicit MetadataOptions parameter to be passed to RunInstances, and i guess the API version of nixops doesn't support that parameter | 01:02:44 |
Arian | * That Imdssupport:V2 Option on the AMI causes an implicit MetadataOptions parameter to be passed to RunInstances when you use the AMi, and i guess the API version of nixops doesn't support that parameter | 01:03:06 |
Arian | The old AMI doesn't have that option set, so there is no implicit unsupported parameter | 01:05:54 |
nh2 | Right. Bit of a brutal error message, The parameter MetadataOptions is not recognized; AWS could at least say it's an implicit parameter added by the Version. | 01:06:05 |
Arian | Yeh this is something somewhere internally going wrong and then that error bubbling up 😅 | 01:07:09 |
Arian | Classic AWS error message | 01:07:23 |
nh2 | Not as good as Microsoft, where on OneDrive, if you download a large folder as a streamed ZIP, they randomly replace files by text files that contain internal error messages whent that machine OOMs, happens in ~30% of files I download from Microsoft | 01:08:19 |
nh2 | The slightly annoying thing here is that this part of NixOps uses boto, not boto3. I'll have a look whether I can get the request to work by convincing that to use a newer Version. Then afterwards I guess I have to see if I can get the thing updated to a newer nixops, or switch it out (it'll be a long work and risky though). | 01:09:00 |
Arian | Hope there are no coredumps in there lol | 01:09:02 |
nh2 | Arian: Do you know if there's a list/changelog of Versions for EC2? | 01:10:22 |
nh2 | Currently checking https://stackoverflow.com/questions/60366935/find-latest-aws-api-version-for-locking-the-version | 01:11:11 |
Arian | https://docs.aws.amazon.com/AWSEC2/latest/APIReference/Query-Requests.html | 01:11:46 |
Arian | The latest version is 2016-11-15 | 01:11:58 |
Arian | They almost never make breaking changes. IMDSv2 was needed because enormous security hole | 01:12:45 |
Arian | But i guess that means updating nixops might be enough? | 01:12:56 |
Arian | Seems the lock file is from 2018 | 01:13:08 |
Arian | https://github.com/boto/botocore/tree/develop/botocore/data/ec2 | 01:15:57 |
nh2 | Alternatively, maybe I should just build my own AMI, and turn off the option there.
Because I don't actually need any specific AMI. It's only needed to boot, the machine, in a reproducible fashion (and ideallly one that doesn't rename all devices after reboot when systemd decides that it's another year again where all network interfaces shall be renamed).
After nixops has deployed and rebooted, the software specified by by nixpkgs pin runs anway. | 01:16:04 |
Arian | Here are all the api versions! | 01:16:05 |
nh2 | In reply to @arianvp:matrix.org https://github.com/NixOS/amis/blob/main/upload-ami/src/upload_ami/upload_ami.py#L173 Version-pinning that here: https://github.com/NixOS/amis/blob/99b494036de3f1418c65b62c8a4197e76c10ad74/upload-ami/src/upload_ami/upload_ami.py#L173 | 01:17:04 |
Arian | Fwiw we ship udev rules that should give all network interfaces alt-names based on the AWS eni id | 01:18:36 |
Arian | Which is stable even across instance types | 01:18:50 |