| 20 Dec 2024 |
 Arian | I also noticed that the fluent bit package is a bit wonky (upstream). They vendor all their c dependencies as source code instead of allowing you to link against packages | 08:44:34 |
 commiterate | It's honestly uglier than the CloudWatch Agent | 18:46:20 |
| commiterate | I'm trying to fix Darwin builds on it right now | 18:46:25 |
| commiterate | Is there a better alternative than fluent-bit with systemd-journald and IMDS support? Vector? | 18:49:28 |
| commiterate | * Is there a better alternative than fluent-bit with systemd-journald and IMDS support? Vector? OTEL Collector? | 18:51:56 |
| commiterate | Vector seems like the most sensible one since it's just as flexible as Fluent-Bit and also performance focused (in Rust). OTEL Collector seems like it's only for OTEL stuff, so spiritually closer to the CloudWatch Agent (also written in Go). | 19:09:05 |
| commiterate | I think OTEL Collector might require using the AWS Distro for OpenTelemtry (ADOT) instead for out-of-the-box AWS support. | 19:10:45 |
| commiterate | ugh ADOT does the same command package -> binary rename thing as amazon-ec2-metadata-mock | 19:20:18 |
| commiterate | https://github.com/aws-observability/aws-otel-collector/blob/15a91838c0f16541c64fafd79ca3ae34b2c309f9/Makefile#L79-L88 | 19:20:48 |
| Arian | opentelemetry-contrib can output otel logs from journald iirc | 19:46:25 |
| Arian | And that is packaged in nixos too | 19:46:42 |
| commiterate | not sure what exporters/sinks/destinations are built in though | 19:46:42 |
| commiterate | this fluent-bit build is hell to fix | 19:47:01 |
| Arian | Alloy also is packaged | 19:47:07 |
| Arian | And definitely supports journald | 19:47:13 |
| Arian | yeh otel-collector can do journald:
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/systemdreceiver | 19:51:56 |
| Arian | what is different between ADOT and open-telemetry-collector-contrib ? | 19:52:36 |
| commiterate | I think it has some extra AWS-specific exporters built in that may not be in OTEL core or OTEL community distributions | 19:52:56 |
| Arian | looking at the go.mod seems `adot doesn't have the systemd receiver | 19:53:11 |
| commiterate | I'm not too sure how plugins work in the OTEL collector world (not sure if you can just add more stuff to PATH or if it must be baked into the OTEL collector binary) | 19:53:38 |
| Arian | okay so CloudWatch has an OTLP endpoint. Sounds simple to set up | 20:10:56 |
| commiterate | I think there might be an escape hook to make fluent-bit not try to compile its vendored dependencies like luajit from scratch | 20:11:34 |
| commiterate | so this might be salvageable | 20:11:45 |
| commiterate | IIRC there's some random issues with CloudWatch and X-Ray not being fully OLTP compliant (mostly X-Ray) | 20:12:16 |
| Arian | https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-OTLPSimplesetup.html | 20:14:20 |
| Arian | full example here | 20:14:28 |
| Arian | but idk from the example how I would create a log stream per instance id | 20:16:09 |
| commiterate | the fluent-bit link against system libs flag is bait, it only works for some things (e.g. rdkafka, luajit) but not others (e.g. onigmo which is their regex engine) | 20:46:08 |
| commiterate | * the fluent-bit link against system libs flag is bait, it only works for some things (e.g. rdkafka, luajit) but not others (e.g. onigmo which is their regex engine dependency) | 20:46:14 |
| commiterate | * the fluent-bit link against system libs cmake flag is bait, it only works for some things (e.g. rdkafka, luajit) but not others (e.g. onigmo which is their regex engine dependency) | 20:46:21 |
