NixOS AWS - Public Room Timeline

	NixOS AWS	79 Members
		20 Servers

Load older messages

Sender	Message	Time
22 Oct 2024
Arian	To quote the Amazon docs: > The latest versions of the AWS CLI and AWS SDKs support IMDSv2. To use IMDSv2, make sure that your EC2 instances have the latest versions of the CLI and SDKs.	00:44:06
Arian	It'd be interesting to look at both `aws EC2 run-instances` and nixops in debug mode to see the exact request. But I'm pretty sure it's gonna be the API version parameter that is causing issues	00:53:28
Arian	* It'd be interesting to look at both `aws EC2 run-instances` and nixops in debug mode to see the exact request differences. But I'm pretty sure it's gonna be the API version parameter that is causing issues	00:53:44
nh2	In reply to @arianvp:matrix.org I thought NixOps is unmaintained. I think they even removed the AWS plugin recently Oh sure it is. Doesn't mean that all the startups using it to manage their infra evaporate though :D With NixOps 2 never getting a real release and NixOps 4 already being the next "when it's done" software (I have some doubts that rewriting in Rust will solve community maintenance; it would certainly make debugging this stuff a lot harder than `pdb` allowed) ... NixOS 1.7 is the latest official stable release (TM) 🙃 Still works pretty well though. I'm checking your replies now, very helpful, one moment	00:53:45
Arian	If updating nixops is not an option I'm afraid you'll have to upload a custom AMI with that setting unset :(	00:57:09
Arian	But that'll get you into trouble as well as the ec2 nixos modules rely on IMDSv2 these days	00:59:38
Arian	Wait nah that's fine	01:00:52
Arian	IMDSv2 works on IMDSv1 images.	01:01:02
Arian	Just not the other way around	01:01:06
nh2	In reply to @arianvp:matrix.org The only hit on google is this one stack overflow issue : https://stackoverflow.com/questions/76495875/cant-provision-ec2-instance-with-ansible Yes, I saw that. My plan is to answer that StackOverflow question eventually. maybe NixOps uses a very old version of the AWS SDK that is maybe incompatible with the AWS API? The key thing is it can't' just be NixOps alone. Because NixOps (and the same curl command) launches the old instance but not the new. So it's not a plain AWS API removal. I suspect it's something like "if you use an AMI that uses new feature X, you need to give some extra launch options". This is the curl (credentials removed): curl https://ec2.eu-west-1.amazonaws.com -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Authorization: AWS4-HMAC-SHA256 Credential=.../20241021/eu-west-1/ec2/aws4_request,SignedHeaders=host;x-amz-date,Signature=...' -H 'X-Amz-Date: 20241021T013814Z' --data 'Action=RunInstances&BlockDeviceMapping.1.DeviceName=%2Fdev%2Fxvda&BlockDeviceMapping.1.Ebs.DeleteOnTermination=true&BlockDeviceMapping.1.Ebs.Encrypted=false&BlockDeviceMapping.1.Ebs.SnapshotId=snap-0e6c56ddc0453c75a&BlockDeviceMapping.1.Ebs.VolumeSize=100&BlockDeviceMapping.1.Ebs.VolumeType=gp3&ClientToken=...&ImageId=ami-0e7d1823ac80520e6&InstanceInitiatedShutdownBehavior=stop&InstanceType=t2.medium&KeyName=my-nixops&MaxCount=1&MinCount=1&SecurityGroup.1=nixops-corp-test-2&UserData=...this only contains the SSH private key in base64...&Version=2014-10-01' -D - Indeed `Version=2014-10-01`. So what you're suggesting is quite likely.	01:01:15
nh2	In reply to @arianvp:matrix.org The only hit on google is this one stack overflow issue : https://stackoverflow.com/questions/76495875/cant-provision-ec2-instance-with-ansible * Yes, I saw that. My plan is to answer that StackOverflow question eventually. maybe NixOps uses a very old version of the AWS SDK that is maybe incompatible with the AWS API? The key thing is it can't' just be NixOps alone. Because NixOps (and the same curl command) launches the old instance but not the new. So it's not a plain AWS API removal. I suspect it's something like "if you use an AMI that uses new feature X, you need to give some extra launch options" (what you're suggesting, as I read along). This is the curl (credentials removed): curl https://ec2.eu-west-1.amazonaws.com -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Authorization: AWS4-HMAC-SHA256 Credential=.../20241021/eu-west-1/ec2/aws4_request,SignedHeaders=host;x-amz-date,Signature=...' -H 'X-Amz-Date: 20241021T013814Z' --data 'Action=RunInstances&BlockDeviceMapping.1.DeviceName=%2Fdev%2Fxvda&BlockDeviceMapping.1.Ebs.DeleteOnTermination=true&BlockDeviceMapping.1.Ebs.Encrypted=false&BlockDeviceMapping.1.Ebs.SnapshotId=snap-0e6c56ddc0453c75a&BlockDeviceMapping.1.Ebs.VolumeSize=100&BlockDeviceMapping.1.Ebs.VolumeType=gp3&ClientToken=...&ImageId=ami-0e7d1823ac80520e6&InstanceInitiatedShutdownBehavior=stop&InstanceType=t2.medium&KeyName=my-nixops&MaxCount=1&MinCount=1&SecurityGroup.1=nixops-corp-test-2&UserData=...this only contains the SSH private key in base64...&Version=2014-10-01' -D - Indeed `Version=2014-10-01`. So what you're suggesting is quite likely.	01:01:54
nh2	So I think you pinpointed it exactly	01:02:06
Arian	In reply to @nh2:matrix.org Yes, I saw that. My plan is to answer that StackOverflow question eventually. maybe NixOps uses a very old version of the AWS SDK that is maybe incompatible with the AWS API? The key thing is it can't' just be NixOps alone. Because NixOps (and the same curl command) launches the old instance but not the new. So it's not a plain AWS API removal. I suspect it's something like "if you use an AMI that uses new feature X, you need to give some extra launch options". This is the curl (credentials removed): curl https://ec2.eu-west-1.amazonaws.com -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Authorization: AWS4-HMAC-SHA256 Credential=.../20241021/eu-west-1/ec2/aws4_request,SignedHeaders=host;x-amz-date,Signature=...' -H 'X-Amz-Date: 20241021T013814Z' --data 'Action=RunInstances&BlockDeviceMapping.1.DeviceName=%2Fdev%2Fxvda&BlockDeviceMapping.1.Ebs.DeleteOnTermination=true&BlockDeviceMapping.1.Ebs.Encrypted=false&BlockDeviceMapping.1.Ebs.SnapshotId=snap-0e6c56ddc0453c75a&BlockDeviceMapping.1.Ebs.VolumeSize=100&BlockDeviceMapping.1.Ebs.VolumeType=gp3&ClientToken=...&ImageId=ami-0e7d1823ac80520e6&InstanceInitiatedShutdownBehavior=stop&InstanceType=t2.medium&KeyName=my-nixops&MaxCount=1&MinCount=1&SecurityGroup.1=nixops-corp-test-2&UserData=...this only contains the SSH private key in base64...&Version=2014-10-01' -D - Indeed `Version=2014-10-01`. So what you're suggesting is quite likely. That Imdssupport:V2 Option on the AMI causes an implicit MetadataOptions parameter to be passed to RunInstances, and i guess the API version of nixops doesn't support that parameter	01:02:44
Arian	* That Imdssupport:V2 Option on the AMI causes an implicit MetadataOptions parameter to be passed to RunInstances when you use the AMi, and i guess the API version of nixops doesn't support that parameter	01:03:06
Arian	The old AMI doesn't have that option set, so there is no implicit unsupported parameter	01:05:54
nh2	Right. Bit of a brutal error message, `The parameter MetadataOptions is not recognized`; AWS could at least say it's an implicit parameter added by the `Version`.	01:06:05
Arian	Yeh this is something somewhere internally going wrong and then that error bubbling up 😅	01:07:09
Arian	Classic AWS error message	01:07:23
nh2	Not as good as Microsoft, where on OneDrive, if you download a large folder as a streamed ZIP, they randomly replace files by text files that contain internal error messages whent that machine OOMs, happens in ~30% of files I download from Microsoft	01:08:19
nh2	The slightly annoying thing here is that this part of NixOps uses `boto`, not `boto3`. I'll have a look whether I can get the request to work by convincing that to use a newer `Version`. Then afterwards I guess I have to see if I can get the thing updated to a newer nixops, or switch it out (it'll be a long work and risky though).	01:09:00
Arian	Hope there are no coredumps in there lol	01:09:02
nh2	Arian: Do you know if there's a list/changelog of `Version`s for EC2?	01:10:22
nh2	Currently checking https://stackoverflow.com/questions/60366935/find-latest-aws-api-version-for-locking-the-version	01:11:11
Arian	https://docs.aws.amazon.com/AWSEC2/latest/APIReference/Query-Requests.html	01:11:46
Arian	The latest version is 2016-11-15	01:11:58
Arian	They almost never make breaking changes. IMDSv2 was needed because enormous security hole	01:12:45
Arian	But i guess that means updating nixops might be enough?	01:12:56
Arian	Seems the lock file is from 2018	01:13:08
Arian	https://github.com/boto/botocore/tree/develop/botocore/data/ec2	01:15:57
nh2	Alternatively, maybe I should just build my own AMI, and turn off the option there. Because I don't actually need any specific AMI. It's only needed to boot, the machine, in a reproducible fashion (and ideallly one that doesn't rename all devices after reboot when systemd decides that it's another year again where all network interfaces shall be renamed). After nixops has deployed and rebooted, the software specified by by nixpkgs pin runs anway.	01:16:04

Show newer messages

Back to Room ListRoom Version: 10