!coeAONBrWyDJnYMbMi:nixos.org

NixOS System Operations

400 Members
About system administration for running NixOS systems in production. Declaratively manage your operations. | Room recommendations: #networking:nixos.org107 Servers

Load older messages


SenderMessageTime
8 Dec 2024
@hexa:lossy.networkhexaawwh gross01:42:15
@hexa:lossy.networkhexathe tmpfiles created symlink is just never updated01:42:22
@hexa:lossy.networkhexawhat the actual fuck01:42:27
@hexa:lossy.networkhexanix just fails at yaml01:43:25
@hexa:lossy.networkhexa * nix just fails hard at yaml01:43:32
@hexa:lossy.networkhexa(╯°□°)╯︵ ┻━┻01:43:42
@hexa:lossy.networkhexaso after fixing the expression and running the linter it reformats the file to01:45:17
@hexa:lossy.networkhexa
namespace: rules
groups:
    - name: postgresql
      rules:
        - alert: PostgresqlCollationMismatch
          expr: |-
            {unit="postgresql.service"} |~ "WARNING:  database \"[^\"]+\" has a collation version mismatch" | regexp "database \"(?<database>[^\"]+)\""
          for: 1m
          labels:
            severity: warning
          annotations:
            summary: PostgreSQL database on {{ $labels.machine }} reports database collations mismatches
01:45:28
@hexa:lossy.networkhexaand of course the loki module has a configFile escape02:31:49
@hexa:lossy.networkhexa
  yaml = pkgs.formats.yaml { };
  rulesFile = pkgs.runCommand "rules.yaml" {
    preferLocalBuild = true;
  } ''
    install -m0700 ${yaml.generate "rules.tmp" rulerConfig} $out
    ${lib.getExe' config.services.loki.package "lokitool"} rules lint $out
  '';
02:41:49
@hexa:lossy.networkhexaso perfectly linted every time02:43:40
@hexa:lossy.networkhexaand yet not alerts02:43:44
@hexa:lossy.networkhexafucking clownshow02:43:47
@hexa:lossy.networkhexaeven with debug logging no interesting ruler logs02:50:08
@hexa:lossy.networkhexaearlier today I added a dummy rule to test if alerting works at all19:34:23
@hexa:lossy.networkhexa
      name = "housekeeping";
      rules = [ {
        alert = "LokiAlwaysFiring";
        expr = "vector(1)";
        annotations.summary = "Rule that always fires to test alert propagation";
      } ];
19:34:46
@hexa:lossy.networkhexadoes not propagate 🤷19:34:59
@hexa:lossy.networkhexa
      ruler = {
        storage = {
          type = "local";
          local.directory = "/etc/loki/rules/";
        };
        rule_path = "/tmp/loki-rules-scratch";
        alertmanager_url = "http://127.0.0.1:9093";
        ring.kvstore.store = "inmemory";
      };
19:35:51
@hexa:lossy.networkhexa so am I holding the ruler wrong? K900 maybe? 19:36:01
@k900:0upti.meK900Hmm 19:40:06
@k900:0upti.meK900Sus 19:40:12
@hexa:lossy.networkhexalike I can't even browse the rules it loads, because that wants a stupid tenant id? 😄 19:43:37
@hexa:lossy.networkhexa
# lokitool rules print --address https://loki.lossy.network --id=1
INFO[0000] no rule groups currently exist for this user 
19:44:29
@hexa:lossy.networkhexa

When configured with auth_enabled: false, Loki uses a single tenant. [...] The single tenant ID will be the string fake.

19:45:11
@hexa:lossy.networkhexa
# lokitool rules print --address https://loki.lossy.network --id=fake
INFO[0000] no rule groups currently exist for this user 
19:45:21
@hexa:lossy.networkhexaso it doesn't even load the rules? 🤔19:45:56
@hexa:lossy.networkhexapretty sure this used to work before 3.019:46:56
@hexa:lossy.networkhexaok, so the rules now need to be below ${local.directory}/<tenant>/20:21:08
@hexa:lossy.networkhexa

caller=ruler.go:576 msg="unable to list rules" err="failed to list rule groups for user fake: failed to list rule group for user fake and namespace rules.yaml: error parsing /etc/loki/rules/fake/rules.yaml: /etc/loki/rules/fake/rules.yaml: yaml: unmarshal errors:\n line 1: field namespace not found in type rulefmt.RuleGroups"

20:21:17
@hexa:lossy.networkhexaand mine are now missing a namespace20:21:31

Show newer messages


Back to Room ListRoom Version: 10