| 19 Nov 2024 |
 Arian | systemctl enable --now works fine on NixOS if the units have no WantedBy in the [Install] section | 18:48:40 |
| Arian | it is just an alias for systemctl start in that case | 18:48:47 |
| Arian | so I actually think these scripts (Even though theyr'e ugly) will just work | 18:49:16 |
| Arian | There is nothing to patdch | 18:49:21 |
 commiterate | I see, I'll swap it out of draft then and put out a call for a maintainer. | 19:13:35 |
| Arian | Need to double check it. But I think it's the case | 19:24:41 |
| commiterate | Opened an issue upstream to request a change to systemd device units.
https://github.com/amazonlinux/amazon-ec2-net-utils/issues/112 | 19:37:45 |
| 20 Nov 2024 |
|  Inayet removed their profile picture. | 00:59:37 |
| 21 Nov 2024 |
| commiterate | CloudWatch Agent is finally in Nixpkgs: https://github.com/NixOS/nixpkgs/pull/337212
Tracker: https://nixpk.gs/pr-tracker.html?pr=337212 | 17:32:28 |
| Arian | In reply to @commiterate:matrix.org
CloudWatch Agent is finally in Nixpkgs: https://github.com/NixOS/nixpkgs/pull/337212
Tracker: https://nixpk.gs/pr-tracker.html?pr=337212 Nice. we can backport this to 24.11 given it's a new package | 18:23:52 |
| 23 Nov 2024 |
| commiterate | Hmm I might need to update it to let people specify paths to the configuration files.
That way people can write their own systemd oneshots which dynamically generate a file at runtime during boot (e.g. getting information from IMDS, SSM Parameter Store, Secrets Manager) instead of having to make 1 VM image per configuration (especially since each VM image is several GBs). | 19:08:09 |
| commiterate | PR: https://github.com/NixOS/nixpkgs/pull/358559 | 20:18:23 |
| commiterate | * Hmm I might need to update it to let people specify paths to the configuration files.
That way people can write their own systemd oneshots which dynamically generate a file at runtime during boot (e.g. getting information from IMDS, SSM Parameter Store, Secrets Manager) instead of having to make 1 VM image per configuration permutation (especially since each VM image is several GBs). | 20:18:54 |
| 24 Nov 2024 |
| commiterate | Hmm actually has a bug since I can't extract the desired run as user at build time. | 19:26:40 |
| commiterate | Fixed, though it means agent.run_as_user in the configuration file is no longer respected (i.e. can't change the user at runtime with a CW config file change) which is fine IMO. | 20:47:27 |
| 25 Nov 2024 |
| commiterate | Arian Any concerns with this Fluent Bit module before I try upstreaming it?
https://github.com/commiterate/nix-fluent-bit
Probably going to use it despite the CW Agent work due to the native systemd-journald support and better processing features. That and I'm a bit hesitant now that I've seen the spaghetti under the hood.
| 06:11:20 |
| commiterate | * Arian Any concerns with this Fluent Bit module before I try adding it to Nixpkgs?
https://github.com/commiterate/nix-fluent-bit
Probably going to use it despite the CW Agent work due to the native systemd-journald support and better processing features. That and I'm a bit hesitant now that I've seen the spaghetti under the hood.
| 06:11:31 |
| 1 Dec 2024 |
 @sielicki:matrix.org | fyi, working on a handful of changes related to AWS and ML:
-
Adding the efa kernel module: https://github.com/NixOS/nixpkgs/pull/360347
-
Adding efa-nv-peermem: https://github.com/NixOS/nixpkgs/pull/360375
-
Adding an updateScript for the out-of-tree ena build and a package bump: https://github.com/NixOS/nixpkgs/pull/360326
I expect a few others before the weekend is over:
-
modifying the libfabric drv to support building with efa and HMEM_CUDA
-
adding and building libnccl-ofi, plus extending nccl so that it uses it
with all of these in place (minus the ENA part which is independent) it should be possible to support multinode ML training on aws with nixos.
| 02:31:59 |
| @sielicki:matrix.org | Arian: any ideas on how to expose this in a module and enable it?
EFA supported instances types are here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html#efa-instance-types
efa-nv-peermem and the nccl/libfabric stuff is only really needed on p3/p4.*/p5.*/p5e.*/p5en.*
| 02:35:40 |
| @sielicki:matrix.org | there's a separate discussion worth having about neuron kmods and software support | 02:36:26 |
| Arian | Given its a kernel module do we need an option? Cant we just add it to the image and have udev load it when needed? | 07:47:06 |
| 4 Dec 2024 |
| Arian | nah this looks pretty good. We could perhaps add more structured module types | 10:26:18 |
| Arian | https://github.com/arianvp/nixos-village/blob/main/nix/modules/fluent-bit.nix | 10:26:43 |
| Arian | by using freeformType = | 10:26:50 |
| Arian | the restartTrigger on user is superfluous | 10:28:03 |
| Arian | pretty sure anything that changes the unit file is a restart trigger | 10:28:16 |
| Arian | also the grace option seems unused | 10:28:50 |
| 6 Dec 2024 |
|  Cees de Groot joined the room. | 16:19:14 |
|  adamcstephens joined the room. | 16:20:58 |
| 10 Dec 2024 |
| commiterate | grace is used to set the systemd unit's shutdown timeout for graceful shutdown. We could technically exclude it but I'd rather have systemd also have a timeout in case Fluent Bit has some bug. | 18:04:05 |
