!coeAONBrWyDJnYMbMi:nixos.org

NixOS System Operations

552 Members
About system administration for running NixOS systems in production. Declaratively manage your operations. | Room recommendations: #networking:nixos.org148 Servers

Load older messages


SenderMessageTime
2 Dec 2024
@funkybooboo:matrix.orgNate Stott joined the room.15:02:17
@b10c:b10c.meb10c joined the room.15:59:12
4 Dec 2024
@kaleocheng:matrix.orgkaleo joined the room.09:36:44
@adam:robins.wtf@adam:robins.wtfNote to future self, it's probably cleaner to remove a garage node prior to redeploying it rather than force removing it afterwards.15:03:24
5 Dec 2024
@shados:nixos.devShados joined the room.01:22:47
@qualytllc:matrix.orgDanny Gerhardt joined the room.09:57:42
7 Dec 2024
@sigmasquadron:matrix.orgFernando Rodrigues joined the room.22:12:38
@hexa:lossy.networkhexaanyone here who happily uses loki with rules and alerting?22:15:06
@hexa:lossy.networkhexa * anyone here who happily uses loki with rules and alerting and considers it useful? 22:30:51
@k900:0upti.meK900 ⚡️ I don't have any alerts in it right now, but I've done that before at $work and yes it is useful IF you know what to alert on 22:35:54
@hexa:lossy.networkhexaI find writing the expressions super tedious22:37:19
@hexa:lossy.networkhexaand since I mostly want to log on events it always follows the same pattern22:38:08
@hexa:lossy.networkhexa
sum by (machine) (count_over_time({...})
  |~ "some match" [5m]
)) >0
22:38:50
@hexa:lossy.networkhexa *
sum by (machine) (count_over_time({...}
  |~ "some match" [5m]
)) >0
22:39:05
@k900:0upti.meK900 ⚡️It depends on what you're setting up alerts for tbh 22:39:16
@hexa:lossy.networkhexayeah, 110% events22:39:25
@hexa:lossy.networkhexanot really rates and whatever they think of examples22:39:46
@k900:0upti.meK900 ⚡️I think you can just drop most of that then and just have the filter as the rule 22:41:22
@k900:0upti.meK900 ⚡️And it'll alert if anything matches 22:41:27
@hexa:lossy.networkhexahuh, ok22:41:45
@hexa:lossy.networkhexathat is kinda the simplicity I was looking for22:42:04
@hexa:lossy.networkhexa
      {unit="postgresql.service"}
        |~ "ALTER DATABASE \\w+ REFRESH COLLATION VERSION"
        | regexp "DATABASE (?<database>[a-zA-Z0-9]+) REFRESH" [5m]
22:42:59
@hexa:lossy.networkhexathat would be one example22:43:03
@hexa:lossy.networkhexa *
      {unit="postgresql.service"}
        |~ "ALTER DATABASE \\w+ REFRESH COLLATION VERSION"
        | regexp "DATABASE (?<database>[a-zA-Z0-9]+) REFRESH"
22:43:13
@k900:0upti.meK900 ⚡️Hm I should steal that 22:43:52
@hexa:lossy.networkhexa😛22:44:28
@hexa:lossy.networkhexanow I wish there was webinterface to browse the state of loki22:52:13
@hexa:lossy.networkhexanot the user facing stuff like grafana, more like what prometheus has22:52:22
@hexa:lossy.networkhexacan't say it is alerting based on that expression22:56:05
@hexa:lossy.networkhexa
[93980] WARNING:  database "template1" has a collation version mismatch
[93980] DETAIL:  The database was created using collation version 2.39, but the operating system provides version 2.40.
[93980] HINT:  Rebuild all objects in this database that use the default collation and run ALTER DATABASE template1 REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.
23:00:16

Show newer messages


Back to Room ListRoom Version: 10