| 24 Sep 2024 |
 commiterate | * We don't have any NixOS module that polls IMDS's autoscaling/target-lifecycle-status for Terminated and issues a shutdown right? | 22:10:02 |
 Arian | Nope. We have something internally at work though. Wanted to generalise it to support systemd-inhibit | 23:06:28 |
| 25 Sep 2024 |
|  @luna-null:matrix.org changed their display name from luna-null to Autumn. | 06:39:11 |
| commiterate | Before I bother another EC2 PM or engineer, you know if instance-initiated shutdown (e.g. shutdown inside the OS) is subject to the EC2-initiated stop/terminate time limits?
E.g. If I let a systemd unit have a 1 hour shutdown timeout, will an instance-initiated shutdown get to wait that full hour or does is it still subject to the ~10 min timeout?
| 17:55:19 |
| Arian | It will die after 10 minutes | 19:29:47 |
| Arian | If you don't want that you need lifecycle hooks | 19:29:55 |
| Arian | And send heart beats | 19:29:59 |
| commiterate | That's the default case for an EC2-initiated shutdown. What the docs aren't clear about is an instance-initiated shutdown (e.g. we poll IMDS's autoscaling/target-lifecycle-status and do an instance-initiated shutdown when we see Terminated as the target state).
They also aren't clear about what they do if the instance goes into terminated before the lifecycle hook timeout and we don't call CompleteLifecycleAction).
| 20:27:34 |
| commiterate | * That's the default case for an EC2-initiated shutdown. What the docs aren't clear about is an instance-initiated shutdown (e.g. we poll IMDS's autoscaling/target-lifecycle-status and do an instance-initiated shutdown by running shutdown when we see Terminated as the target state).
They also aren't clear about what they do if the instance goes into terminated before the lifecycle hook timeout and we don't call CompleteLifecycleAction).
| 20:27:50 |
| commiterate | * That's the default case for an EC2-initiated shutdown. What the docs aren't clear about is an instance-initiated shutdown (e.g. we poll IMDS's autoscaling/target-lifecycle-status and do an instance-initiated shutdown by running shutdown when we see Terminated as the target state).
They also aren't clear about what they do if the instance goes into terminated before the lifecycle hook timeout and we don't call CompleteLifecycleAction.
| 20:36:50 |
| Arian | I think for instance instanitated shutdown it's just the Lifecycle hook timeout that matters | 21:11:02 |
| Arian | Like systemd doesn't send any signal to the hardware when you call systemctl poweroff. It will only do the poweroff after all the services actually stopped | 21:11:34 |
| Arian | If you have a systemd inhibit or a service that just takes an hour to shut down. There is no way for EC2 to know the machine is shutting down | 21:12:07 |
| Arian | So the machine will be shut down my ec2 after the lifecycle hook timeout | 21:12:45 |
| commiterate | kk. I'm checking with ASG and EC2 just in case but that's what I'm suspecting as well. | 21:52:46 |
| 1 Oct 2024 |
|  -_o joined the room. | 21:03:49 |
| 6 Oct 2024 |
| Arian | I have a WIP new image builder that uses grub in hybrid mode | 16:16:23 |
| Arian | So we can have 1 image for EFI and BIOS | 16:16:31 |
| Arian | Eventually I wanna drop bios completely. But too many image types still around that don't support nitro.. | 16:17:09 |
| 10 Oct 2024 |
| Arian | I'm in a bit of a pickle. So I want to stop uploading legacy-bios AMIs and always have a EFI System Partition. (So the bootmode of the AMIs will change to uefi-preferred).
However existing EC2 instances that were booted with an old AMI will not have an ESP... So I can't outright remove the code.
| 12:50:16 |
| Arian | I guess I can make it depend on stateVersion but it's so ugly... However forcing people to create their AWS instances to update to 24.11 seems harsh. | 12:50:43 |
|  p4cmanus3r joined the room. | 13:25:15 |
| 11 Oct 2024 |
| commiterate | Would it matter given that to use the new AMI, they're either replacing the root volume or launching a new instance? | 02:33:55 |
| commiterate | * Would it matter given that to use the new AMI, they're either replacing the root volume (which has both the ESP and root partitions) or launching a new instance? | 02:39:19 |
| Arian | If you use a new AMI it is no problem | 06:29:24 |
| Arian | Problem is people doing nixos-rebuild switch on an existing instance | 06:29:51 |
| Arian | commiterate: do you know if we need the AWS networkd stuff for EC2 instance connect endpoint? | 13:32:35 |
| Arian | IIRC it works by adding a new ENI to the instance. I don't think our AMI manages that ENI so idk if it works out of the box | 13:32:53 |
| Arian | ah nope just works! sick | 19:46:55 |
| commiterate | In reply to @arianvp:matrix.org
Problem is people doing nixos-rebuild switch on an existing instance Wait are the AMIs tracking the ami repo for something?
If they nixos-rebuild switch, I don't see what would go awry if they're just tracking the normal NixOS channels like any regular user.
| 21:55:02 |
