| 23 Nov 2024 |
 commiterate | PR: https://github.com/NixOS/nixpkgs/pull/358559 | 20:18:23 |
| commiterate | * Hmm I might need to update it to let people specify paths to the configuration files.
That way people can write their own systemd oneshots which dynamically generate a file at runtime during boot (e.g. getting information from IMDS, SSM Parameter Store, Secrets Manager) instead of having to make 1 VM image per configuration permutation (especially since each VM image is several GBs). | 20:18:54 |
| 24 Nov 2024 |
| commiterate | Hmm actually has a bug since I can't extract the desired run as user at build time. | 19:26:40 |
| commiterate | Fixed, though it means agent.run_as_user in the configuration file is no longer respected (i.e. can't change the user at runtime with a CW config file change) which is fine IMO. | 20:47:27 |
| 25 Nov 2024 |
| commiterate | Arian Any concerns with this Fluent Bit module before I try upstreaming it?
https://github.com/commiterate/nix-fluent-bit
Probably going to use it despite the CW Agent work due to the native systemd-journald support and better processing features. That and I'm a bit hesitant now that I've seen the spaghetti under the hood.
| 06:11:20 |
| commiterate | * Arian Any concerns with this Fluent Bit module before I try adding it to Nixpkgs?
https://github.com/commiterate/nix-fluent-bit
Probably going to use it despite the CW Agent work due to the native systemd-journald support and better processing features. That and I'm a bit hesitant now that I've seen the spaghetti under the hood.
| 06:11:31 |
| 1 Dec 2024 |
 @sielicki:matrix.org | fyi, working on a handful of changes related to AWS and ML:
-
Adding the efa kernel module: https://github.com/NixOS/nixpkgs/pull/360347
-
Adding efa-nv-peermem: https://github.com/NixOS/nixpkgs/pull/360375
-
Adding an updateScript for the out-of-tree ena build and a package bump: https://github.com/NixOS/nixpkgs/pull/360326
I expect a few others before the weekend is over:
-
modifying the libfabric drv to support building with efa and HMEM_CUDA
-
adding and building libnccl-ofi, plus extending nccl so that it uses it
with all of these in place (minus the ENA part which is independent) it should be possible to support multinode ML training on aws with nixos.
| 02:31:59 |
| @sielicki:matrix.org | Arian: any ideas on how to expose this in a module and enable it?
EFA supported instances types are here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html#efa-instance-types
efa-nv-peermem and the nccl/libfabric stuff is only really needed on p3/p4.*/p5.*/p5e.*/p5en.*
| 02:35:40 |
| @sielicki:matrix.org | there's a separate discussion worth having about neuron kmods and software support | 02:36:26 |
 Arian | Given its a kernel module do we need an option? Cant we just add it to the image and have udev load it when needed? | 07:47:06 |
| 4 Dec 2024 |
| Arian | nah this looks pretty good. We could perhaps add more structured module types | 10:26:18 |
| Arian | https://github.com/arianvp/nixos-village/blob/main/nix/modules/fluent-bit.nix | 10:26:43 |
| Arian | by using freeformType = | 10:26:50 |
| Arian | the restartTrigger on user is superfluous | 10:28:03 |
| Arian | pretty sure anything that changes the unit file is a restart trigger | 10:28:16 |
| Arian | also the grace option seems unused | 10:28:50 |
| 6 Dec 2024 |
|  Cees de Groot joined the room. | 16:19:14 |
|  @adam:robins.wtf joined the room. | 16:20:58 |
| 10 Dec 2024 |
| commiterate | grace is used to set the systemd unit's shutdown timeout for graceful shutdown. We could technically exclude it but I'd rather have systemd also have a timeout in case Fluent Bit has some bug. | 18:04:05 |
| commiterate | As for adding more structural typing to the config options, it seems like a maintenance burden since we need to keep up with any config schema changes on the Fluent Bit side. | 18:05:06 |
| commiterate | * grace is used to set the systemd unit's shutdown timeout for graceful shutdown. We could technically exclude it but I'd rather have systemd also have a timeout in case Fluent Bit has some bug.
We also need it anyways since systemd has a default of 90s.
https://www.freedesktop.org/software/systemd/man/latest/systemd-system.conf.html#DefaultTimeoutStartSec=
| 18:08:35 |
| commiterate | * grace is used to set the systemd unit's shutdown timeout for graceful shutdown.
We need it since systemd has a default of 90s.
https://www.freedesktop.org/software/systemd/man/latest/systemd-system.conf.html#DefaultTimeoutStartSec=
| 19:16:17 |
| 13 Dec 2024 |
| @sielicki:matrix.org | yes and no -- similar to ENA it's a question of whether the in-tree module should be preferred to the out of tree one. | 04:04:18 |
| 15 Dec 2024 |
| commiterate | Should we find a maintainer and try to merge the net-utils Nix package as is (since it seems like it's fine for now) or should we try to submit changes upstream to swap to systemd device units first? | 03:25:59 |
| 16 Dec 2024 |
| commiterate | PR: https://github.com/NixOS/nixpkgs/pull/365493 | 04:43:35 |
| commiterate | As a related bit, adding more IMDS categories to the built-in AWS filter plugin: https://github.com/fluent/fluent-bit/pull/9727 | 04:45:04 |
| commiterate | * Related note: adding more IMDS categories to the built-in AWS filter plugin: https://github.com/fluent/fluent-bit/pull/9727 | 04:45:17 |
| Arian | I mean. We can unconditionally include the EFA module in the AWS config I mean | 12:40:24 |
| Arian | https://github.com/NixOS/nixpkgs/pull/365690 | 21:20:24 |
| 20 Dec 2024 |
| commiterate | Looks like the fluent-bit package has effectively no maintainers. | 03:45:54 |
