| 27 Mar 2025 |
 Arian | @commiterate:matrix.org: do you remember why you added zstd as a buildInput to fluent-bit?
Fluent-bit started vendoring libzstd-1.5.7 in their tree but because we add zstd to buildInputs and nixos has a different version it somehow ends up dynamically linking to libzstd-1.5.6 but with the wrong headers. This causes fluent-bit to segfault | 22:34:54 |
| Arian | I hate fluent-bit | 22:35:05 |
| 28 Mar 2025 |
| Arian | Okay... Fluent-bit just segfaults even without this stuff.. argh | 00:24:45 |
| Arian | It's due to libsystemd dlopening libzstd | 00:25:04 |
 Ilan Joselevich (Kranzes) | In reply to @arianvp:matrix.org
Okay... Fluent-bit just segfaults even without this stuff.. argh Go sleep, we going surfing tomorrow morning | 00:28:54 |
 commiterate | It's needed by rdkafka. Otherwise build complains that the library isn't available.
https://github.com/NixOS/nixpkgs/blob/698214a32beb4f4c8e3942372c694f40848b360d/pkgs/by-name/fl/fluent-bit/package.nix#L86-L87
| 03:52:31 |
| commiterate | The way Fluent Bit does things is extremely cursed. | 03:52:54 |
| commiterate | It was a pretty terrible time getting the package to work properly. | 03:53:24 |
| commiterate | * It's needed by rdkafka. Otherwise build complained that the library isn't available.
https://github.com/NixOS/nixpkgs/blob/698214a32beb4f4c8e3942372c694f40848b360d/pkgs/by-name/fl/fluent-bit/package.nix#L86-L87
| 03:55:10 |
| commiterate | I didn't think it would be this brittle though. | 03:56:12 |
| commiterate | I feel like at some point they're just going to vendor their entire dependency graph and continue having 0 instructions on what's needed to build it, so the only working build setup will be theirs and people can only consume their pre-built binaries. | 03:59:10 |
| Arian | So the problem even occurs when I use all their vendores libraries | 08:17:00 |
| Arian | Problem is that libsystemd dlopens an incompatible libzstd into memory | 08:17:21 |
| Arian | And then all hell breaks loose | 08:17:27 |
| Arian | 🔥 | 08:17:45 |
| commiterate | Can't say I'm surprised with their lack of systemd tests in CI. Did this not fail the fluent-bit NixOS test? | 14:12:02 |
| commiterate | * It feels like at some point they're just going to vendor their entire dependency graph and continue having 0 instructions on what's needed to build it, so the only working build setup will be theirs and people can only consume their pre-built binaries. | 14:12:27 |
 geekodour | i am finding it hard to find teams using aws+nix if any of your teams are hiring remotely, pls let me know! much appreciate! | 14:12:59 |
| commiterate |
- https://hydra.nixos.org/job/nixos/trunk-combined/nixos.tests.fluent-bit.x86_64-linux
- https://hydra.nixos.org/job/nixos/trunk-combined/nixos.tests.fluent-bit.aarch64-linux
It's passing with 3.2.9 so seems like the tests aren't triggering the libzstd load (the failures have no logs so I'm assuming those were cancelled evals).
| 14:17:10 |
| commiterate | *
- https://hydra.nixos.org/job/nixos/trunk-combined/nixos.tests.fluent-bit.x86_64-linux
- https://hydra.nixos.org/job/nixos/trunk-combined/nixos.tests.fluent-bit.aarch64-linux
It's passing with 3.2.9 so seems like the tests aren't triggering the libzstd load (the failures have no logs so I'm assuming those were cancelled evals).
Not sure if it's only loaded when pushing logs to a remote sink.
| 14:48:59 |
| Arian | Yep! That's because libzstd code path only triggers with log messages larger than 512 bytes | 15:49:21 |
| Arian | I have a nixos test that triggers the issue locally | 15:49:29 |
| Arian | In reply to @geekodour:matrix.org
i am finding it hard to find teams using aws+nix if any of your teams are hiring remotely, pls let me know! much appreciate! https://job-boards.greenhouse.io/mercury/jobs/5457761004 | 15:50:30 |
| Arian | This is my team | 15:50:37 |
| Arian | I still need to make the test failure reliable though. It only fails sometimes. I guess the crash doesn't always happen due to some randomization | 17:06:57 |
| geekodour | In reply to @arianvp:matrix.org
This is my team great! i sent my application in. it'll be coming from gigs@geekodour dot org. lmk if you find it interesting! | 17:48:24 |
| 4 Apr 2025 |
|  Phil Hale joined the room. | 17:18:39 |
| 11 Apr 2025 |
| Arian | @commiterate:matrix.org: seems Amazon started shipping the go binary of EC2 instance connect to Linux as well | 11:18:15 |
| Arian | But I can't find the source anywhere. Which is weird. As Amazon linux is supposed to be open source | 11:18:27 |
| commiterate | it's probably stuck in the open source review process | 18:52:40 |
