| 7 Dec 2024 |
|  SigmaSquadron joined the room. | 22:12:38 |
 hexa | anyone here who happily uses loki with rules and alerting? | 22:15:06 |
| hexa | * anyone here who happily uses loki with rules and alerting and considers it useful? | 22:30:51 |
 K900 | I don't have any alerts in it right now, but I've done that before at $work and yes it is useful IF you know what to alert on | 22:35:54 |
| hexa | I find writing the expressions super tedious | 22:37:19 |
| hexa | and since I mostly want to log on events it always follows the same pattern | 22:38:08 |
| hexa | sum by (machine) (count_over_time({...})
|~ "some match" [5m]
)) >0
| 22:38:50 |
| hexa | * sum by (machine) (count_over_time({...}
|~ "some match" [5m]
)) >0
| 22:39:05 |
| K900 | It depends on what you're setting up alerts for tbh | 22:39:16 |
| hexa | yeah, 110% events | 22:39:25 |
| hexa | not really rates and whatever they think of examples | 22:39:46 |
| K900 | I think you can just drop most of that then and just have the filter as the rule | 22:41:22 |
| K900 | And it'll alert if anything matches | 22:41:27 |
| hexa | huh, ok | 22:41:45 |
| hexa | that is kinda the simplicity I was looking for | 22:42:04 |
| hexa | {unit="postgresql.service"}
|~ "ALTER DATABASE \\w+ REFRESH COLLATION VERSION"
| regexp "DATABASE (?<database>[a-zA-Z0-9]+) REFRESH" [5m]
| 22:42:59 |
| hexa | that would be one example | 22:43:03 |
| hexa | * {unit="postgresql.service"}
|~ "ALTER DATABASE \\w+ REFRESH COLLATION VERSION"
| regexp "DATABASE (?<database>[a-zA-Z0-9]+) REFRESH"
| 22:43:13 |
| K900 | Hm I should steal that | 22:43:52 |
| hexa | 😛 | 22:44:28 |
| hexa | now I wish there was webinterface to browse the state of loki | 22:52:13 |
| hexa | not the user facing stuff like grafana, more like what prometheus has | 22:52:22 |
| hexa | can't say it is alerting based on that expression | 22:56:05 |
| hexa | [93980] WARNING: database "template1" has a collation version mismatch
[93980] DETAIL: The database was created using collation version 2.39, but the operating system provides version 2.40.
[93980] HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE template1 REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.
| 23:00:16 |
| hexa | rulerConfig = {
groups = [ {
name = "postgresql";
rules = [ {
alert = "PostgresqlCollationMismatch";
expr = ''
{unit="postgresql.service"}
|~ 'WARNING: database "[^"]+" has a collation version mismatch'
| regexp 'database "(?<database>[^"]+)"'
'';
labels.severity = "warning";
annotations.summary = "PostgreSQL database {{ $labels.database }} on {{ $labels.machine }}i reports database collation mismatch";
} ];
} ];
};
| 23:01:48 |
| hexa | * rulerConfig = {
groups = [ {
name = "postgresql";
rules = [ {
alert = "PostgresqlCollationMismatch";
expr = ''
{unit="postgresql.service"}
|~ 'WARNING: database "[^"]+" has a collation version mismatch'
| regexp 'database "(?<database>[^"]+)"'
'';
labels.severity = "warning";
annotations.summary = "PostgreSQL database {{ $labels.database }} on {{ $labels.machine }}i reports database collation mismatch";
} ];
} ];
};
| 23:01:52 |
| hexa | so for one it doesn't like single quotes 😄 | 23:03:24 |
| hexa | and doesn't complain about it | 23:03:28 |
| hexa | * rulerConfig = {
groups = [ {
name = "postgresql";
rules = [ {
alert = "PostgresqlCollationMismatch";
expr = ''
{unit="postgresql.service"}
|~ "WARNING: database \"[^\"]+\" has a collation version mismatch"
| regexp "database \"(?<database>[^\"]+)\""
'';
labels.severity = "warning";
annotations.summary = "PostgreSQL database {{ $labels.database }} on {{ $labels.machine }}i reports database collation mismatch";
} ];
} ];
};
| 23:04:17 |
| 8 Dec 2024 |
 adamcstephens | My favorite, when something silently accepts invalid input | 00:22:52 |
