!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

468 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.147 Servers

Load older messages


SenderMessageTime
27 Jun 2026
@vcunat:matrix.orgvcunat

And if I substitute "random" stuff I get

ERROR: new row for relation "builds" violates check constraint "builds_check"

07:37:38
@vcunat:matrix.orgvcunat

Without setting much stuff it's like

hydra=# BEGIN;
BEGIN
hydra=*# EXPLAIN ANALYZE  UPDATE builds SET finished = 1 where id=123456;
                                                          QUERY PLAN                                                          
------------------------------------------------------------------------------------------------------------------------------
 Update on builds  (cost=0.57..2.79 rows=0 width=0) (actual time=4.086..4.087 rows=0.00 loops=1)
   Buffers: shared hit=79 read=18 dirtied=10
   ->  Index Scan using builds_pkey on builds  (cost=0.57..2.79 rows=1 width=10) (actual time=0.293..0.294 rows=1.00 loops=1)
         Index Cond: (id = 123456)
         Index Searches: 1
         Buffers: shared hit=3 read=2
 Planning:
   Buffers: shared hit=294
 Planning Time: 1.508 ms
 Trigger nrbuildsfinished: time=0.648 calls=1
 Execution Time: 4.811 ms
(11 rows)

hydra=*# ROLLBACK;
ROLLBACK
07:38:38
@vcunat:matrix.orgvcunat🤔 https://grafana.nixos.org/d/rrbV5fdik/postgres-node07:48:31
@vcunat:matrix.orgvcunatLongest transaction: 3.38 days 😱07:48:46
@vcunat:matrix.orgvcunatWe have abnormal amount of rollbacks today, apparently.07:50:45
@vcunat:matrix.orgvcunat* And we have abnormal amount of rollbacks today, apparently.07:50:48
@vcunat:matrix.orgvcunatHigh "longest transaction" seems normal historically.07:51:40
@vcunat:matrix.orgvcunat(though it feels risky)07:51:56
@hexa:lossy.networkhexa (signing key rotation when)do you have a grafana account?08:12:43
@grimmauld:m.grimmauld.deGrimmauld (any/all) i don't, i think. I have a github account with committer access, and community builder access. That's about it. What would i need to do to get a grafana account? 08:17:35
@hexa:lossy.networkhexa (signing key rotation when)I think we have open registration?08:17:54
@hexa:lossy.networkhexa (signing key rotation when)you just need write permissions after that08:19:12
@grimmauld:m.grimmauld.deGrimmauld (any/all)looks like there is a signup, does someone have to approve that or how does that work?08:19:44
@hexa:lossy.networkhexa (signing key rotation when)no, just do08:22:34
@hexa:lossy.networkhexa (signing key rotation when)sure can08:24:32
@grimmauld:m.grimmauld.deGrimmauld (any/all) alright, profile done. And sorry, i lost internet connectivity for a bit and your message appeared after mine despite sending order being the other way round 08:26:51
@grimmauld:m.grimmauld.deGrimmauld (any/all)image.png
Download image.png
08:26:51
@grimmauld:m.grimmauld.deGrimmauld (any/all)* alright, account done. And sorry, i lost internet connectivity for a bit and your message appeared after mine despite sending order being the other way round08:27:11
@hexa:lossy.networkhexa (signing key rotation when)already granted editor08:27:27
@grimmauld:m.grimmauld.deGrimmauld (any/all)https://grafana.nixos.org/d/mel9fqb/very-simple-combined-occupancy yep that worked09:00:26
@grimmauld:m.grimmauld.deGrimmauld (any/all)I'll also see whether i can correlate load with occupancy, if we have metrics for that. Might be interesting too.09:07:34
@hexa:lossy.networkhexa (signing key rotation when)yeah, the database doesn't look outrageous in total09:23:27
@hexa:lossy.networkhexa (signing key rotation when)I think missing runnables is really in the queue-runner09:23:35
@hexa:lossy.networkhexa (signing key rotation when)especially with so many steps it has ingested09:26:36
@defelo:matrix.defelo.deDefelo joined the room.10:46:59
@joerg:thalheim.ioMic92 hexa (signing key rotation when): https://github.com/NixOS/infra/pull/1099 My recommendation is to first deploy this on staging, eval nixos unstable small and if logs are clear, we can promote this to prod. I have rebased it now 11:14:29
@joerg:thalheim.ioMic92 Once this is in I can run perf again and see what the new bottleneck is. 11:14:49
@hexa:lossy.networkhexa (signing key rotation when)applying to staging-hydra11:15:35
@joerg:thalheim.ioMic92I don't think the current bottlenecks will be very visible in prometheus. But they are very obvious if you open htop11:16:50
@grimmauld:m.grimmauld.deGrimmauld (any/all) Can't really open htop without being infra team though. And prometheus does show some info worth looking at imo, even if it doesn't show the reasons behind metrics. 11:18:16

Show newer messages


Back to Room ListRoom Version: 6