 | NixOS AWS | 64 Members |
| 15 Servers |
| NixOS AWS | 64 Members |
| 15 Servers |
| Sender | Message | Time |
|---|---|---|
| 22 Oct 2024 | ||
 Arian | https://github.com/NixOS/amis/blob/main/upload-ami/src/upload_ami/upload_ami.py#L173 | 00:17:53 |
| Arian | Hmm actually I think maybe NixOps uses a very old version of the AWS SDK that is maybe incompatible with the AWS API? | 00:19:26 |
| Arian | Can you show me the raw curl command? | 00:19:43 |
| Arian | The only hit on google is this one stack overflow issue : https://stackoverflow.com/questions/76495875/cant-provision-ec2-instance-with-ansible | 00:23:03 |
| Arian | You're not on EC2 classic or something weird right? (The old VPC-less EC2) | 00:25:59 |
| Arian | So yeh because the AMI sets that option, i think that causes an implicit MetadataOptions to be passed. However IMDSv2 is new. So maybe you're using a very old SDK. The EC2 API is versioned (See the Version parameter in the curl request) | 00:36:06 |
| Arian | So maybe if you are trying to spawn an instance with IMDSv2 enabled using the older API version you get this error | 00:36:35 |
| Arian | Advice here is: update boto3 | 00:39:34 |
| Arian | * Advice here is: update boto3 dependency of nixops | 00:39:44 |
| Arian | To quote the Amazon docs: > The latest versions of the AWS CLI and AWS SDKs support IMDSv2. To use IMDSv2, make sure that your EC2 instances have the latest versions of the CLI and SDKs. | 00:44:06 |
| Arian | It'd be interesting to look at both `aws EC2 run-instances` and nixops in debug mode to see the exact request. But I'm pretty sure it's gonna be the API version parameter that is causing issues | 00:53:28 |
| Arian | * It'd be interesting to look at both `aws EC2 run-instances` and nixops in debug mode to see the exact request differences. But I'm pretty sure it's gonna be the API version parameter that is causing issues | 00:53:44 |
 nh2 | In reply to @arianvp:matrix.org Oh sure it is. Doesn't mean that all the startups using it to manage their infra evaporate though :D I'm checking your replies now, very helpful, one moment | 00:53:45 |
| Arian | If updating nixops is not an option I'm afraid you'll have to upload a custom AMI with that setting unset :( | 00:57:09 |
| Arian | But that'll get you into trouble as well as the ec2 nixos modules rely on IMDSv2 these days | 00:59:38 |
| Arian | Wait nah that's fine | 01:00:52 |
| Arian | IMDSv2 works on IMDSv1 images. | 01:01:02 |
| Arian | Just not the other way around | 01:01:06 |
| nh2 | In reply to @arianvp:matrix.org Yes, I saw that. My plan is to answer that StackOverflow question eventually.
The key thing is it can't' just be NixOps alone. Because NixOps (and the same curl command) launches the old instance but not the new. So it's not a plain AWS API removal. This is the curl (credentials removed): Indeed | 01:01:15 |
| nh2 | In reply to @arianvp:matrix.org* Yes, I saw that. My plan is to answer that StackOverflow question eventually.
The key thing is it can't' just be NixOps alone. Because NixOps (and the same curl command) launches the old instance but not the new. So it's not a plain AWS API removal. This is the curl (credentials removed): Indeed | 01:01:54 |
| nh2 | So I think you pinpointed it exactly | 01:02:06 |
| Arian | In reply to @nh2:matrix.orgThat Imdssupport:V2 Option on the AMI causes an implicit MetadataOptions parameter to be passed to RunInstances, and i guess the API version of nixops doesn't support that parameter | 01:02:44 |
| Arian | * That Imdssupport:V2 Option on the AMI causes an implicit MetadataOptions parameter to be passed to RunInstances when you use the AMi, and i guess the API version of nixops doesn't support that parameter | 01:03:06 |
| Arian | The old AMI doesn't have that option set, so there is no implicit unsupported parameter | 01:05:54 |
| nh2 | Right. Bit of a brutal error message, The parameter MetadataOptions is not recognized; AWS could at least say it's an implicit parameter added by the Version. | 01:06:05 |
| Arian | Yeh this is something somewhere internally going wrong and then that error bubbling up 😅 | 01:07:09 |
| Arian | Classic AWS error message | 01:07:23 |
| nh2 | Not as good as Microsoft, where on OneDrive, if you download a large folder as a streamed ZIP, they randomly replace files by text files that contain internal error messages whent that machine OOMs, happens in ~30% of files I download from Microsoft | 01:08:19 |
| nh2 | The slightly annoying thing here is that this part of NixOps uses boto, not boto3.I'll have a look whether I can get the request to work by convincing that to use a newer Version.Then afterwards I guess I have to see if I can get the thing updated to a newer nixops, or switch it out (it'll be a long work and risky though). | 01:09:00 |
| Arian | Hope there are no coredumps in there lol | 01:09:02 |