!atvIbxHoEqNcAIxYpN:nixos.org

NixOS AWS

65 Members
17 Servers

You have reached the beginning of time (for this room).


SenderMessageTime
6 May 2025
@commiterate:matrix.orgcommiterate *

We use Nix + system-manager to bake reproducible Amazon Linux 2023 AMIs. There's a shell script snippet in this GitHub issue: https://github.com/aws/ec2-image-builder-roadmap/issues/110

# Switch from ssm-user to the default user.
sudo su ec2-user

# Install RPM packages.
sudo dnf install --assumeyes curl-minimal git

# Install Nix.
curl --fail --location https://install.determinate.systems/nix/tag/v3.1.1 --proto '=https' --show-error --silent --tlsv1.2 | sh -s -- install --no-confirm
. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh

# Setup Nix flake registry.
nix registry add nixpkgs github:NixOS/nixpkgs/{Git revision hash}
nix registry add system-manager github:numtide/system-manager/{Git revision hash}

# Install Nix packages.
nix profile install system-manager

# Apply system-manager configuration (installs system-wide packages and sets up systemd units).
sudo $(command -v system-manager) pre-populate --flake 'git+{Git HTTPs URL}&rev={Git revision hash}#{system-manager flake output key}'

CloudFormation templates are generated with the AWS CDK. The infrastructure code essentially:

  • Locks an Amazon-managed AMI ID from their SSM public parameters into the cdk.context.json file.
  • Calls nix flake metadata to get the flake's Git hash and construct a flake URL for the flake itself and some inputs needed in the script (we run Nix eval on a system in a VPC with company network connectivity).
  • Generates an SSM document with the aforementioned script.
  • Sets up the EC2 Image Builder infrastructure that auto-builds an AMI on CloudFormation stack deploys.
  • Reference the AMI in a launch template which is then used in an auto-scaling group.
  • Use CloudFormation rolling update to bounce the auto-scaling group (until ASG instance refresh is supported in CloudFormation).

Once the CloudFormation image import situation is improved, we'll move the non-bootstrap stuff to use NixOS disk images created with the systemd-repart helpers.

This ends up being fully reproducible because Amazon Linux 2023 locks the Amazon Linux package repository version (these are globally versioned now), so any Nix dependencies like curl and git are locked by the base Amazon Linux 2023 AMI itself.

18:49:07
@commiterate:matrix.orgcommiterate *

We use Nix + system-manager to bake reproducible Amazon Linux 2023 AMIs. There's a shell script snippet in this GitHub issue: https://github.com/aws/ec2-image-builder-roadmap/issues/110

# Switch from ssm-user to the default user.
sudo su ec2-user

# Install RPM packages.
sudo dnf install --assumeyes curl-minimal git

# Install Nix.
curl --fail --location https://install.determinate.systems/nix/tag/v3.1.1 --proto '=https' --show-error --silent --tlsv1.2 | sh -s -- install --no-confirm
. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh

# Setup Nix flake registry.
nix registry add nixpkgs github:NixOS/nixpkgs/{Git revision hash}
nix registry add system-manager github:numtide/system-manager/{Git revision hash}

# Install Nix packages.
nix profile install system-manager

# Apply system-manager configuration (installs system-wide packages and sets up systemd units).
sudo $(command -v system-manager) pre-populate --flake 'git+{Git HTTPs URL}&rev={Git revision hash}#{system-manager flake output key}'

CloudFormation templates are generated with the AWS CDK. The infrastructure code essentially:

  • Locks an Amazon-managed AMI ID from their SSM public parameters into the cdk.context.json file.
  • Calls nix flake metadata to get the flake's Git hash and construct a flake URL for the flake itself and some inputs needed in the script (we run Nix eval on a system in a VPC with company network connectivity).
  • Generates an SSM document with the aforementioned script.
  • Sets up the EC2 Image Builder infrastructure that auto-builds an AMI on CloudFormation stack deploys.
  • Reference the AMI in a launch template which is then used in an auto-scaling group.
  • Use CloudFormation rolling update to bounce the auto-scaling group (until ASG instance refresh is supported in CloudFormation).

Once the CloudFormation EC2 disk image import situation is improved, we'll move the non-bootstrap stuff to use NixOS disk images created with the systemd-repart helpers.

This ends up being fully reproducible because Amazon Linux 2023 locks the Amazon Linux package repository version (these are globally versioned now), so any Nix dependencies like curl and git are locked by the base Amazon Linux 2023 AMI itself.

18:55:50
@adam:robins.wtfadamcstephensWe recently migrated to NixOS for our application servers, and have a couple more instance profiles before we'll be fully on NixOS. We're deploying Elixir applications with an in-house deployment tool, leveraging S3 for deployment coordination and as a binary cache. We build Nix paths for NixOS and applications separately, push them to the cache and write those paths to a bucket along with other metadata including the git sha. Then during deploy we pull and activate those paths (no eval). While we do build our own AMI, we also use this same path-based deployment through amazon-init to switch to the correct profile on boot.18:56:00
@adam:robins.wtfadamcstephensI work for a company named CalmWave. We're a US-based healthcare startup.18:56:36
7 May 2025
@randoneering:matrix.orgjustin@randoneering joined the room.00:35:19
15 May 2025
@commiterate:matrix.orgcommiterate* NVIDIA (though I'm on the DGX Cloud side, not the GPU side. There's some internal Nix users pushing for better NixOS NVIDIA driver + CUDA support though)07:33:09
18 May 2025
@arianvp:matrix.orgArian

urgh:

https://github.com/aws/amazon-ec2-metadata-mock/issues/234

12:14:29
@commiterate:matrix.orgcommiterateTurnaround on most of the AWS open source stuff outside of the AWS CDK or SDKs (which have dedicated support engineers) is pretty terrible.17:26:05
@commiterate:matrix.orgcommiterate* Turnaround on most of the AWS open source stuff outside of the AWS CDK or CLI + SDKs (which have dedicated support engineers) is pretty terrible.17:26:16
@geekodour:matrix.orggeekodour i am quite confused about them keeping the github actions totally community supported 17:33:42
@commiterate:matrix.orgcommiterateMost AWS teams are fairly small (maybe 5-10 people) and there's generally no distinction between engineering, QA, and devops/SRE. It's just "software development engineer" (SDE) which handles all 3 functions which they can get away with because the internal tooling is, IMO, very well done. Each organization is basically a big ensemble of a bunch of small teams (e.g. control plane team, data plane team, frontend team for AWS console stuff) as the leaves which roll up into a reporting tree. That means most teams are actually pretty lean and don't have much extra capacity for other stuff. More important areas like the AWS CLI + SDKs or AWS CDK will have additional dedicated resources beyond SDEs to handle a lot of customer interactions.19:30:44
@commiterate:matrix.orgcommiterate

I don't know if the IMDS team themselves own EC2 metadata mock. If it is, they're probably dealing with a lot of internal IMDS development work.

If not, it's probably something more owned by solutions architects (e.g. most stuff under the aws-labs GitHub org like coldsnap) who are also working on many other things.

19:32:34

Show newer messages


Back to Room ListRoom Version: 10