8 Dec 2024 |
hexa | awwh gross | 01:42:15 |
hexa | the tmpfiles created symlink is just never updated | 01:42:22 |
hexa | what the actual fuck | 01:42:27 |
hexa | nix just fails at yaml | 01:43:25 |
hexa | * nix just fails hard at yaml | 01:43:32 |
hexa | (╯°□°)╯︵ ┻━┻ | 01:43:42 |
hexa | so after fixing the expression and running the linter it reformats the file to | 01:45:17 |
hexa | namespace: rules
groups:
- name: postgresql
rules:
- alert: PostgresqlCollationMismatch
expr: |-
{unit="postgresql.service"} |~ "WARNING: database \"[^\"]+\" has a collation version mismatch" | regexp "database \"(?<database>[^\"]+)\""
for: 1m
labels:
severity: warning
annotations:
summary: PostgreSQL database on {{ $labels.machine }} reports database collations mismatches
| 01:45:28 |
hexa | and of course the loki module has a configFile escape | 02:31:49 |
hexa | yaml = pkgs.formats.yaml { };
rulesFile = pkgs.runCommand "rules.yaml" {
preferLocalBuild = true;
} ''
install -m0700 ${yaml.generate "rules.tmp" rulerConfig} $out
${lib.getExe' config.services.loki.package "lokitool"} rules lint $out
'';
| 02:41:49 |
hexa | so perfectly linted every time | 02:43:40 |
hexa | and yet not alerts | 02:43:44 |
hexa | fucking clownshow | 02:43:47 |
hexa | even with debug logging no interesting ruler logs | 02:50:08 |
hexa | earlier today I added a dummy rule to test if alerting works at all | 19:34:23 |
hexa | name = "housekeeping";
rules = [ {
alert = "LokiAlwaysFiring";
expr = "vector(1)";
annotations.summary = "Rule that always fires to test alert propagation";
} ];
| 19:34:46 |
hexa | does not propagate 🤷 | 19:34:59 |
hexa | ruler = {
storage = {
type = "local";
local.directory = "/etc/loki/rules/";
};
rule_path = "/tmp/loki-rules-scratch";
alertmanager_url = "http://127.0.0.1:9093";
ring.kvstore.store = "inmemory";
};
| 19:35:51 |
hexa | so am I holding the ruler wrong? K900 maybe? | 19:36:01 |
K900 | Hmm | 19:40:06 |
K900 | Sus | 19:40:12 |
hexa | like I can't even browse the rules it loads, because that wants a stupid tenant id? 😄 | 19:43:37 |
hexa | # lokitool rules print --address https://loki.lossy.network --id=1
INFO[0000] no rule groups currently exist for this user
| 19:44:29 |
hexa |
When configured with auth_enabled: false, Loki uses a single tenant. [...] The single tenant ID will be the string fake.
| 19:45:11 |
hexa | # lokitool rules print --address https://loki.lossy.network --id=fake
INFO[0000] no rule groups currently exist for this user
| 19:45:21 |
hexa | so it doesn't even load the rules? 🤔 | 19:45:56 |
hexa | pretty sure this used to work before 3.0 | 19:46:56 |
hexa | ok, so the rules now need to be below ${local.directory}/<tenant>/ | 20:21:08 |
hexa |
caller=ruler.go:576 msg="unable to list rules" err="failed to list rule groups for user fake: failed to list rule group for user fake and namespace rules.yaml: error parsing /etc/loki/rules/fake/rules.yaml: /etc/loki/rules/fake/rules.yaml: yaml: unmarshal errors:\n line 1: field namespace not found in type rulefmt.RuleGroups"
| 20:21:17 |
hexa | and mine are now missing a namespace | 20:21:31 |