| 8 Dec 2024 |
 hexa | Redacted or Malformed Event | 20:26:17 |
| hexa | so the linter is now happy with my rules file but loki isn't | 20:28:02 |
| hexa | ok, so the linter adds a namespace key to the file and loki does not like it | 20:35:35 |
| hexa | clownshow | 20:35:37 |
| hexa | K900: getting back to your expression recommendation now | 20:36:03 |
| hexa | rule_name=PostgresqlCollationMismatch rule_type=alerting query="{unit=\"postgresql.service\"} |~ \"database \\\"[^\\\"]+\\\" has a collation version mismatch\" | regexp \"database \\\"(?<database>[^\\\"]+)\\\"\"" query_hash=2906991201 msg="rule result is not a vector or scalar" err=null
| 20:36:21 |
| hexa |
rule result is not a vector or scalar
| 20:36:33 |
| hexa | 
Download image.png | 20:37:52 |
| hexa | at least that works now | 20:37:56 |
| hexa | so count_over_time is always implied for event logging | 20:46:13 |
 adamcstephens | i guess that makes sense? | 22:33:45 |
| hexa | yeah, k900 implied it wasn't yesterday fwiw | 22:42:28 |
| hexa | which would make queries much more easy to write in the face of how badly the feedback of the ruler is | 22:43:06 |
| hexa | at least I'm now getting fast feedback | 22:46:27 |
| hexa | building '/nix/store/3f7ww8pjj8v32w45vcflz6l7lvc767vp-rules.yaml.drv'...
ERRO[0000] unable parse rules file error="could not parse expression for alert 'PostgresqlCollationMismatch' in group 'postgresql': parse error at line 5, col 1: syntax error: unexpected $end, expecting )" file=/nix/store/b90hjjsvamndv0z2a9vqfz91w66lbc43-rules.yaml
lokitool: error: prepare operation unsuccessful, unable to parse rules files: file read error, try --help
error: builder for '/nix/store/3f7ww8pjj8v32w45vcflz6l7lvc767vp-rules.yaml.drv' failed with exit code 1;
last 2 log lines:
> ERRO[0000] unable parse rules file error="could not parse expression for alert 'PostgresqlCollationMismatch' in group 'postgresql': parse error at line 5, col 1: syntax error: unexpected $end, expecting )" file=/nix/store/b90hjjsvamndv0z2a9vqfz91w66lbc43-rules.yaml
> lokitool: error: prepare operation unsuccessful, unable to parse rules files: file read error, try --help
For full logs, run 'nix log /nix/store/3f7ww8pjj8v32w45vcflz6l7lvc767vp-rules.yaml.drv'.
| 22:46:30 |
| hexa | finally | 22:50:39 |
| hexa | 
Download image.png | 22:50:39 |
| adamcstephens | Nice | 23:57:10 |
| adamcstephens | I definitely need that collation mismatch alert ;) | 23:57:41 |
| 9 Dec 2024 |
| hexa | pay up! 😄 | 00:00:13 |
| adamcstephens | i gotta set up loki first | 00:00:28 |
| hexa | man, I hate how shitty log based alerting is | 00:00:31 |
| hexa | it could be so simple afer all | 00:00:39 |
| hexa | * it could be so simple after all | 00:00:42 |
| hexa | e.g. logwatch | 00:00:53 |
| adamcstephens | what's the whole, working, rule look like? | 00:03:01 |
| hexa | count_over_time({unit="postgresql.service"}
|~ "database \"[^\"]+\" has a collation version mismatch"
| regexp "database \"(?<database>[^\"]+)\""
[5m])
| 00:03:47 |
 magic_rb | In reply to @hexa:lossy.network
sent an image. Alerts through matrix? Bold move | 00:04:14 |
| hexa | falls back to pushover when synapse or the bot are not up | 00:04:52 |
| hexa | that is why alertmanager support routing 🤷 | 00:05:11 |
