| 8 Dec 2024 |
 hexa | the rules are just not as robust ans simple to test as promql | 00:49:26 |
| hexa | # lokitool rules lint ruler.yml
ERRO[0000] unable parse rules file error="could not parse expression for alert 'PostgresqlCollationMismatch' in group 'postgresql': parse error at line 1, col 19: syntax error: unexpected IDENTIFIER" file=ruler.yml
lokitool: error: prepare operation unsuccessful, unable to parse rules files: file read error, try --help
| 00:50:15 |
| hexa | I also love how builtins.toJSON puts everything on line 1 | 00:50:36 |
| hexa | because that is totally for the benefit of dealing with such an error | 00:50:46 |
 magic_rb | Kind of wish there was a way to make toJSON spew out pretty printed json | 00:56:06 |
| hexa | the loki module should provide an option to accept rules directly and lint them | 01:22:20 |
| hexa | {
"groups": [
{
"name": "postgresql",
"rules": [
{
"alert": "PostgresqlCollationMismatch",
"annotations": {
"summary": "PostgreSQL database on {{ $labels.machine }} reports database collations mismatches"
},
"expr": "sum by (machine) (counter_over_time({unit=\"postgresql.service\"} |~ \"REFRESH COLLATION VERSION\" [5m])) > 0\n",
"for": "1m",
"labels": {
"severity": "warning"
}
}
]
}
]
}
| 01:39:44 |
| hexa | ERRO[0000] unable parse rules file error="could not parse expression for alert 'PostgresqlCollationMismatch' in group 'postgresql': parse error at line 1, col 19: syntax error: unexpected IDENTIFIER" file=/tmp/rules.yml
| 01:39:48 |
| hexa | really, line 1 again? | 01:39:52 |
| hexa | also line 1 when remarshalled to yaml | 01:40:37 |
| hexa |
| 01:41:18 |
| hexa | * groups:
- name: postgresql
rules:
- alert: PostgresqlCollationMismatch
annotations:
summary: PostgreSQL database on {{ $labels.machine }} reports database collations
mismatches
expr: "sum by (machine) (counter_over_time({unit=\"postgresql.service\"} |~ \"\
REFRESH COLLATION VERSION\" [5m])) > 0\n"
for: 1m
labels:
severity: warning
| 01:41:23 |
| hexa | that yaml is also the shit | 01:41:28 |
| hexa | awwh gross | 01:42:15 |
| hexa | the tmpfiles created symlink is just never updated | 01:42:22 |
| hexa | what the actual fuck | 01:42:27 |
| hexa | nix just fails at yaml | 01:43:25 |
| hexa | * nix just fails hard at yaml | 01:43:32 |
| hexa | (╯°□°）╯︵ ┻━┻ | 01:43:42 |
| hexa | so after fixing the expression and running the linter it reformats the file to | 01:45:17 |
| hexa | namespace: rules
groups:
- name: postgresql
rules:
- alert: PostgresqlCollationMismatch
expr: |-
{unit="postgresql.service"} |~ "WARNING: database \"[^\"]+\" has a collation version mismatch" | regexp "database \"(?<database>[^\"]+)\""
for: 1m
labels:
severity: warning
annotations:
summary: PostgreSQL database on {{ $labels.machine }} reports database collations mismatches
| 01:45:28 |
| hexa | and of course the loki module has a configFile escape | 02:31:49 |
| hexa | yaml = pkgs.formats.yaml { };
rulesFile = pkgs.runCommand "rules.yaml" {
preferLocalBuild = true;
} ''
install -m0700 ${yaml.generate "rules.tmp" rulerConfig} $out
${lib.getExe' config.services.loki.package "lokitool"} rules lint $out
'';
| 02:41:49 |
| hexa | so perfectly linted every time | 02:43:40 |
| hexa | and yet not alerts | 02:43:44 |
| hexa | fucking clownshow | 02:43:47 |
| hexa | even with debug logging no interesting ruler logs | 02:50:08 |
| hexa | earlier today I added a dummy rule to test if alerting works at all | 19:34:23 |
| hexa | name = "housekeeping";
rules = [ {
alert = "LokiAlwaysFiring";
expr = "vector(1)";
annotations.summary = "Rule that always fires to test alert propagation";
} ];
| 19:34:46 |
| hexa | does not propagate 🤷 | 19:34:59 |
