| 27 Jun 2026 |
vcunat | And if I substitute "random" stuff I get
ERROR: new row for relation "builds" violates check constraint "builds_check"
| 07:37:38 |
vcunat | Without setting much stuff it's like
hydra=# BEGIN;
BEGIN
hydra=*# EXPLAIN ANALYZE UPDATE builds SET finished = 1 where id=123456;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------
Update on builds (cost=0.57..2.79 rows=0 width=0) (actual time=4.086..4.087 rows=0.00 loops=1)
Buffers: shared hit=79 read=18 dirtied=10
-> Index Scan using builds_pkey on builds (cost=0.57..2.79 rows=1 width=10) (actual time=0.293..0.294 rows=1.00 loops=1)
Index Cond: (id = 123456)
Index Searches: 1
Buffers: shared hit=3 read=2
Planning:
Buffers: shared hit=294
Planning Time: 1.508 ms
Trigger nrbuildsfinished: time=0.648 calls=1
Execution Time: 4.811 ms
(11 rows)
hydra=*# ROLLBACK;
ROLLBACK
| 07:38:38 |
vcunat | 🤔 https://grafana.nixos.org/d/rrbV5fdik/postgres-node | 07:48:31 |
vcunat | Longest transaction: 3.38 days 😱 | 07:48:46 |
vcunat | We have abnormal amount of rollbacks today, apparently. | 07:50:45 |
vcunat | * And we have abnormal amount of rollbacks today, apparently. | 07:50:48 |
vcunat | High "longest transaction" seems normal historically. | 07:51:40 |
vcunat | (though it feels risky) | 07:51:56 |
hexa (signing key rotation when) | do you have a grafana account? | 08:12:43 |
Grimmauld (any/all) | i don't, i think. I have a github account with committer access, and community builder access. That's about it. What would i need to do to get a grafana account? | 08:17:35 |
hexa (signing key rotation when) | I think we have open registration? | 08:17:54 |
hexa (signing key rotation when) | you just need write permissions after that | 08:19:12 |
Grimmauld (any/all) | looks like there is a signup, does someone have to approve that or how does that work? | 08:19:44 |
hexa (signing key rotation when) | no, just do | 08:22:34 |
hexa (signing key rotation when) | sure can | 08:24:32 |
Grimmauld (any/all) | alright, profile done. And sorry, i lost internet connectivity for a bit and your message appeared after mine despite sending order being the other way round | 08:26:51 |
Grimmauld (any/all) |  Download image.png | 08:26:51 |
Grimmauld (any/all) | * alright, account done. And sorry, i lost internet connectivity for a bit and your message appeared after mine despite sending order being the other way round | 08:27:11 |
hexa (signing key rotation when) | already granted editor | 08:27:27 |
Grimmauld (any/all) | https://grafana.nixos.org/d/mel9fqb/very-simple-combined-occupancy
yep that worked | 09:00:26 |
Grimmauld (any/all) | I'll also see whether i can correlate load with occupancy, if we have metrics for that. Might be interesting too. | 09:07:34 |
hexa (signing key rotation when) | yeah, the database doesn't look outrageous in total | 09:23:27 |
hexa (signing key rotation when) | I think missing runnables is really in the queue-runner | 09:23:35 |
hexa (signing key rotation when) | especially with so many steps it has ingested | 09:26:36 |
| Defelo joined the room. | 10:46:59 |
Mic92 | hexa (signing key rotation when): https://github.com/NixOS/infra/pull/1099 My recommendation is to first deploy this on staging, eval nixos unstable small and if logs are clear, we can promote this to prod. I have rebased it now | 11:14:29 |
Mic92 | Once this is in I can run perf again and see what the new bottleneck is. | 11:14:49 |
hexa (signing key rotation when) | applying to staging-hydra | 11:15:35 |
Mic92 | I don't think the current bottlenecks will be very visible in prometheus. But they are very obvious if you open htop | 11:16:50 |
Grimmauld (any/all) | Can't really open htop without being infra team though. And prometheus does show some info worth looking at imo, even if it doesn't show the reasons behind metrics. | 11:18:16 |