| 23 May 2024 |
|  samrose joined the room. | 20:11:34 |
| 27 May 2024 |
|  Harrison Katz joined the room. | 15:07:23 |
| Harrison Katz | Hello! Arian We're trying to debug that AMI networking issue from Friday. If you're online today and would like to chat back and forth we would be much appreciative. I'll also be online tomorrow (or any day this week) to chat. Thanks! | 15:08:22 |
| 30 May 2024 |
|  o1lo0lol1o joined the room. | 14:54:06 |
| o1lo0lol1o | What's the recommended way to get journald logs to cloudwatch? | 14:54:32 |
 Arian | I can tell you what we do at work | 15:34:19 |
| Arian | We use fluent-bit with the following config:
[INPUT]
name systemd
tag journald
[FILTER]
name aws
match journald
imds_version v2
account_id true
[OUTPUT]
name cloudwatch_logs
match journald
region us-east-1
log_group_name /ec2/logs
log_stream_template $ec2_instance_id
log_stream_name default
auto_create_group true
| 15:37:03 |
| Arian | There is no fluent-bit NixOS module yet. We have one internally. | 15:37:17 |
| Arian | * We use fluent-bit with the following config:
[INPUT]
name systemd
tag journald
[FILTER]
name aws
match journald
imds_version v2
account_id true
[OUTPUT]
name cloudwatch_logs
match journald
region us-east-1
log_group_name /ec2/logs
log_stream_template $ec2_instance_id
auto_create_group true
| 15:39:07 |
| Arian | * We use fluent-bit with the following config:
[INPUT]
name systemd
tag journald
db ''${STATE_DIRECTORY}/journald.db
[FILTER]
name aws
match journald
imds_version v2
account_id true
[OUTPUT]
name cloudwatch_logs
match journald
region us-east-1
log_group_name /ec2/logs
log_stream_template $ec2_instance_id
log_stream_name default
auto_create_group true
| 15:42:13 |
| Arian | I can recommend it. zero issues so far :) | 15:44:11 |
| o1lo0lol1o | So you just made a systemd service that runs https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/misc/fluent-bit/default.nix | 15:52:39 |
| o1lo0lol1o | Where does that config go? | 15:52:50 |
| Arian | pass it as —config_file option to the fluent-bit binary
| 16:05:18 |
| 5 Jun 2024 |
|  reepicheep42 joined the room. | 20:26:06 |
| 11 Jun 2024 |
|  TimekillerTK joined the room. | 09:56:16 |
| 22 Jun 2024 |
|  geekodour joined the room. | 16:21:37 |
| geekodour | Redacted or Malformed Event | 16:22:32 |
| 28 Jun 2024 |
 @AleXoundOS:matrix.org | Encountered a situation, when userdata script did not start:
Jun 28 11:34:56 ip-10-0-96-47.eu-central-1.compute.internal systemd[1]: Reached target Multi-User System.
Jun 28 11:34:56 ip-10-0-96-47.eu-central-1.compute.internal systemd[1]: Starting Reconfigure the system from EC2 userdata on startup...
Jun 28 11:34:56 ip-10-0-96-47.eu-central-1.compute.internal amazon-init-start[978]: attempting to fetch configuration from EC2 user data...
Jun 28 11:34:56 ip-10-0-96-47.eu-central-1.compute.internal amazon-init-start[978]: /nix/store/gzx4ihb8hdxl1yj18kfhm5an6vvz59mn-unit-script-amazon-init-start/bin/amazon-init-start: line 16: /etc/ec2-metadata/user-data: No such file or directory
Jun 28 11:34:56 ip-10-0-96-47.eu-central-1.compute.internal amazon-init-start[978]: no user data is available
Jun 28 11:34:56 ip-10-0-96-47.eu-central-1.compute.internal systemd[1]: Finished Reconfigure the system from EC2 userdata on startup.
But with manualy reboot it worked:
Jun 28 11:40:34 ip-10-0-96-47.eu-central-1.compute.internal systemd[1]: Reached target Multi-User System.
Jun 28 11:40:34 ip-10-0-96-47.eu-central-1.compute.internal systemd[1]: Starting Reconfigure the system from EC2 userdata on startup...
Jun 28 11:40:34 ip-10-0-96-47.eu-central-1.compute.internal amazon-init-start[934]: attempting to fetch configuration from EC2 user data...
Jun 28 11:40:34 ip-10-0-96-47.eu-central-1.compute.internal amazon-init-start[934]: running EC2 User Data bootstrap
| 11:44:48 |
| @AleXoundOS:matrix.org | * Encountered a situation, when userdata script did not start:
Jun 28 11:34:56 ip-10-0-96-47.eu-central-1.compute.internal systemd[1]: Reached target Multi-User System.
Jun 28 11:34:56 ip-10-0-96-47.eu-central-1.compute.internal systemd[1]: Starting Reconfigure the system from EC2 userdata on startup...
Jun 28 11:34:56 ip-10-0-96-47.eu-central-1.compute.internal amazon-init-start[978]: attempting to fetch configuration from EC2 user data...
Jun 28 11:34:56 ip-10-0-96-47.eu-central-1.compute.internal amazon-init-start[978]: /nix/store/gzx4ihb8hdxl1yj18kfhm5an6vvz59mn-unit-script-amazon-init-start/bin/amazon-init-start: line 16: /etc/ec2-metadata/user-data: No such file or directory
Jun 28 11:34:56 ip-10-0-96-47.eu-central-1.compute.internal amazon-init-start[978]: no user data is available
Jun 28 11:34:56 ip-10-0-96-47.eu-central-1.compute.internal systemd[1]: Finished Reconfigure the system from EC2 userdata on startup.
But with manual reboot it worked:
Jun 28 11:40:34 ip-10-0-96-47.eu-central-1.compute.internal systemd[1]: Reached target Multi-User System.
Jun 28 11:40:34 ip-10-0-96-47.eu-central-1.compute.internal systemd[1]: Starting Reconfigure the system from EC2 userdata on startup...
Jun 28 11:40:34 ip-10-0-96-47.eu-central-1.compute.internal amazon-init-start[934]: attempting to fetch configuration from EC2 user data...
Jun 28 11:40:34 ip-10-0-96-47.eu-central-1.compute.internal amazon-init-start[934]: running EC2 User Data bootstrap
| 11:45:15 |
| @AleXoundOS:matrix.org | It happened only for the first time among ~100 boots. | 11:46:10 |
| 1 Jul 2024 |
| Arian | Do you have ipv6 enabled? | 08:46:30 |
| @AleXoundOS:matrix.org | Yes, I have IPv6 enabled:
# ifconfig
enX0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.117.211 netmask 255.255.128.0 broadcast 10.0.127.255
inet6 fe80::7b:b7ff:fe6c:ef85 prefixlen 64 scopeid 0x20<link>
ether 02:7b:b7:6c:ef:85 txqueuelen 1000 (Ethernet)
RX packets 15051562 bytes 2684395427 (2.5 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 9503900 bytes 7788542100 (7.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
| 08:49:07 |
| Arian | Yeh so there is a race condition in the ec2 metadata script | 21:33:42 |
| Arian | Network-online.target is either reached when ipv6 or ipv4 is ready and our script only reaches out on the metadata server on ipv4 | 21:34:18 |
| Arian | And then fails and crashes | 21:34:24 |
| Arian | That ifconfig suggests ipv6 is disabled
| 21:34:57 |
| Arian | You don't have an ipv6 address assigned | 21:35:09 |
| Arian | So you must have run into something else | 21:35:22 |
| @AleXoundOS:matrix.org | In reply to @arianvp:matrix.org
That ifconfig suggests ipv6 is disabled
(I meant it's enabled in Linux kernel.) | 21:37:21 |
