| 27 Jun 2026 |
 whispers [& it/fae] | * | 06:05:29 |
| whispers [& it/fae] | * | 06:06:48 |
| whispers [& it/fae] | * | 06:09:43 |
 Vladimír Čunát | Looks like very slow ingestion right now. | 06:51:26 |
| Vladimír Čunát | (no idea why) | 06:51:32 |
| Vladimír Čunát | And it prefers nixpkgs/unstable which has no linux builds left. | 06:52:15 |
| Vladimír Čunát | Though we also have builds in nixos/*-small 🤔 That seems weird. | 06:52:55 |
| Vladimír Čunát | Ah, wrong assumptions. In the 7h old eval there are 1.2k linux builds left. | 07:00:21 |
| Vladimír Čunát | Let me try restarting the queue runner 🤷 Seems low-risk. | 07:01:18 |
| Vladimír Čunát | This morning the runner is logging lots of
sqlx::query: slow statement: execution time exceeded alert threshold
| 07:10:20 |
 K900 | Does it log which query? | 07:13:07 |
| Vladimír Čunát | Typical example was
Jun 27 06:29:42 mimas hydra-queue-runner[884148]: 2026-06-27T06:29:42.410544Z WARN request:complete_build:succeed_step_by_uuid:succeed_step:mark_succeeded_build:update_build: sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE builds SET finished …" db.statement="\n\n\n UPDATE builds SET\n finished = 1,\n buildStatus = $2,\n startTime = $3,\n stopTime = $4,\n size = $5,\n closureSize = $6,\n releaseName = $7,\n isCachedBuild = $8,\n notificationPendingSince = $4\n WHERE\n id = $1\n" rows_affected=1 rows_returned=0 elapsed=136.6054209s elapsed_secs=136.6054209 slow_threshold=1s method=POST uri=/runner.v1.RunnerService/CompleteBuild version=HTTP/2.0 machine_id="2d917310-f886-420b-b961-6709654457ba" build_id="ea683ccd-60b5-49cc-b2db-0fc9dcec66f4" machine_id=2d917310-f886-420b-b961-6709654457ba build_id=ea683ccd-60b5-49cc-b2db-0fc9dcec66f4 machine_id=2d917310-f886-420b-b961-6709654457ba drv_path=hb4wbdx8jknqb2jgp23jxn1nbd03ag7c-steel-language-server-0.8.2.drv build_id=333158552
| 07:15:18 |
| K900 | Hmmm | 07:15:56 |
| K900 | This is sus | 07:15:59 |
| Vladimír Čunát | After the queue-runner restart it's mainly get_queued_builds:get_not_finished_builds | 07:16:02 |
| K900 | Can you explain analyze those on the real db? | 07:16:33 |
| Vladimír Čunát | It's really extreme now. Over 15 minutes it did 20 build steps. | 07:17:34 |
| Vladimír Čunát | Though I get that it can have a slower start. | 07:17:50 |
| Vladimír Čunát | It doesn't show a usable SQL, does it? | 07:30:18 |
| Vladimír Čunát | All those $1, $2, etc. | 07:30:35 |
| Vladimír Čunát | And if I substitute "random" stuff I get
ERROR: new row for relation "builds" violates check constraint "builds_check"
| 07:37:38 |
| Vladimír Čunát | Without setting much stuff it's like
hydra=# BEGIN;
BEGIN
hydra=*# EXPLAIN ANALYZE UPDATE builds SET finished = 1 where id=123456;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------
Update on builds (cost=0.57..2.79 rows=0 width=0) (actual time=4.086..4.087 rows=0.00 loops=1)
Buffers: shared hit=79 read=18 dirtied=10
-> Index Scan using builds_pkey on builds (cost=0.57..2.79 rows=1 width=10) (actual time=0.293..0.294 rows=1.00 loops=1)
Index Cond: (id = 123456)
Index Searches: 1
Buffers: shared hit=3 read=2
Planning:
Buffers: shared hit=294
Planning Time: 1.508 ms
Trigger nrbuildsfinished: time=0.648 calls=1
Execution Time: 4.811 ms
(11 rows)
hydra=*# ROLLBACK;
ROLLBACK
| 07:38:38 |
| Vladimír Čunát | 🤔 https://grafana.nixos.org/d/rrbV5fdik/postgres-node | 07:48:31 |
| Vladimír Čunát | Longest transaction: 3.38 days 😱 | 07:48:46 |
| Vladimír Čunát | We have abnormal amount of rollbacks today, apparently. | 07:50:45 |
| Vladimír Čunát | * And we have abnormal amount of rollbacks today, apparently. | 07:50:48 |
| Vladimír Čunát | High "longest transaction" seems normal historically. | 07:51:40 |
| Vladimír Čunát | (though it feels risky) | 07:51:56 |
 hexa | do you have a grafana account? | 08:12:43 |
 Grimmauld (any/all) | i don't, i think. I have a github account with committer access, and community builder access. That's about it. What would i need to do to get a grafana account? | 08:17:35 |
