!RROtHmAaQIkiJzJZZE:nixos.org

NixOS Infrastructure

478 Members
Next Infra call: 2024-07-11, 18:00 CEST (UTC+2) | Infra operational issues backlog: https://github.com/orgs/NixOS/projects/52 | See #infra-alerts:nixos.org for real time alerts from Prometheus.149 Servers

Load older messages


SenderMessageTime
7 Jul 2026
@vcunat:matrix.orgVladimír ČunátThis eval has been running for ~3.5h now (instead of usual <1h) and this time it does seem to keep doing something all the time: https://hydra.nixos.org/jobset/nixpkgs/staging-next-26.05 🤕 Maybe that's related to overall slowness which I see (running job count, web UI latency, etc.)09:16:42
@hexa:lossy.networkhexa
   pid   | usename | datname | application_name | client_addr | state  |          query_start          |    duration     | wait_event_type | wait_event |                                                                                                               query                                                                                                                
---------+---------+---------+------------------+-------------+--------+-------------------------------+-----------------+-----------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 2883506 | hydra   | hydra   | hydra-server     | 10.0.40.2   | active | 2026-07-07 09:09:03.753775+00 | 00:12:07.250573 |                 |            | SELECT me.id, me.finished, me.timestamp, me.stoptime, me.jobset_id, me.job, me.nixname, me.system, me.buildstatus, me.releasename FROM builds me WHERE ( ( finished = $1 AND me.jobset_id = $2 ) ) ORDER BY stoptime DESC LIMIT $3
 2873792 | hydra   | hydra   | hydra-server     | 10.0.40.2   | active | 2026-07-07 09:09:10.72368+00  | 00:12:00.280668 |                 |            | SELECT me.id, me.finished, me.timestamp, me.stoptime, me.jobset_id, me.job, me.nixname, me.system, me.buildstatus, me.releasename FROM builds me WHERE ( ( finished = $1 AND me.jobset_id = $2 ) ) ORDER BY stoptime DESC LIMIT $3
 2883919 | hydra   | hydra   | hydra-server     | 10.0.40.2   | active | 2026-07-07 09:09:13.184196+00 | 00:11:57.820152 |                 |            | SELECT me.id, me.finished, me.timestamp, me.stoptime, me.jobset_id, me.job, me.nixname, me.system, me.buildstatus, me.releasename FROM builds me WHERE ( ( finished = $1 AND me.jobset_id = $2 ) ) ORDER BY stoptime DESC LIMIT $3
 2880936 | hydra   | hydra   | hydra-server     | 10.0.40.2   | active | 2026-07-07 09:09:28.558658+00 | 00:11:42.44569  |                 |            | SELECT me.id, me.finished, me.timestamp, me.stoptime, me.jobset_id, me.job, me.nixname, me.system, me.buildstatus, me.releasename FROM builds me WHERE ( ( finished = $1 AND me.jobset_id = $2 ) ) ORDER BY stoptime DESC LIMIT $3
 2882318 | hydra   | hydra   | hydra-server     | 10.0.40.2   | active | 2026-07-07 09:09:43.869771+00 | 00:11:27.134577 |                 |            | SELECT me.id, me.finished, me.timestamp, me.stoptime, me.jobset_id, me.job, me.nixname, me.system, me.buildstatus, me.releasename FROM builds me WHERE ( ( finished = $1 AND me.jobset_id = $2 ) ) ORDER BY stoptime DESC LIMIT $3
 2886527 | hydra   | hydra   | hydra-server     | 10.0.40.2   | active | 2026-07-07 09:10:06.176828+00 | 00:11:04.82752  |                 |            | SELECT me.id, me.finished, me.timestamp, me.stoptime, me.jobset_id, me.job, me.nixname, me.system, me.buildstatus, me.releasename FROM builds me WHERE ( ( finished = $1 AND me.jobset_id = $2 ) ) ORDER BY stoptime DESC LIMIT $3
 2875052 | hydra   | hydra   | hydra-server     | 10.0.40.2   | active | 2026-07-07 09:10:25.579051+00 | 00:10:45.425297 |                 |            | SELECT me.id, me.finished, me.timestamp, me.stoptime, me.jobset_id, me.job, me.nixname, me.system, me.buildstatus, me.releasename FROM builds me WHERE ( ( finished = $1 AND me.jobset_id = $2 ) ) ORDER BY stoptime DESC LIMIT $3
 2888877 | hydra   | hydra   | hydra-server     | 10.0.40.2   | active | 2026-07-07 09:10:49.780288+00 | 00:10:21.22406  |                 |            | SELECT me.id, me.finished, me.timestamp, me.stoptime, me.jobset_id, me.job, me.nixname, me.system, me.buildstatus, me.releasename FROM builds me WHERE ( ( finished = $1 AND me.jobset_id = $2 ) ) ORDER BY stoptime DESC LIMIT $3
 2927864 | hydra   | hydra   | hydra-server     | 10.0.40.2   | active | 2026-07-07 09:21:10.981574+00 | 00:00:00.022774 |                 |            | SELECT MAX( me.shortrevlength ) FROM jobsetevalinputs me WHERE ( uri = $1 )
 2930630 | hydra   | hydra   | hydra-server     |             | active | 2026-07-07 09:21:10.981574+00 | 00:00:00.022774 |                 |            | SELECT MAX( me.shortrevlength ) FROM jobsetevalinputs me WHERE ( uri = $1 )
 2930631 | hydra   | hydra   | hydra-server     |             | active | 2026-07-07 09:21:10.981574+00 | 00:00:00.022774 |                 |            | SELECT MAX( me.shortrevlength ) FROM jobsetevalinputs me WHERE ( uri = $1 )
 2930632 | hydra   | hydra   | hydra-server     |             | active | 2026-07-07 09:21:10.981574+00 | 00:00:00.022774 |                 |            | SELECT MAX( me.shortrevlength ) FROM jobsetevalinputs me WHERE ( uri = $1 )
 2930629 | hydra   | hydra   | hydra-server     |             | active | 2026-07-07 09:21:10.981574+00 | 00:00:00.022774 |                 |            | SELECT MAX( me.shortrevlength ) FROM jobsetevalinputs me WHERE ( uri = $1 )
09:21:19
@hexa:lossy.networkhexastuck queries since 10 minutes09:21:30
@hexa:lossy.networkhexaRedacted or Malformed Event09:21:38
@vcunat:matrix.orgVladimír ČunátThat eval now died with a deadlock.10:02:15
@vcunat:matrix.orgVladimír Čunát(which is a relatively normal occurrence)10:02:25
@hexa:lossy.networkhexa
hydra=# EXPLAIN (ANALYZE, BUFFERS)
SELECT
  id, finished, timestamp, stoptime, jobset_id, job, nixname,
  system, buildstatus, releasename
FROM builds
WHERE finished = 0
  AND jobset_id = 123
ORDER BY stoptime DESC
LIMIT 50;
                                                                        QUERY PLAN                                                                         
-----------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.57..14185.53 rows=50 width=120) (actual time=114.857..114.858 rows=0.00 loops=1)
   Buffers: shared hit=78753
   ->  Index Scan using indexbuildsonfinishedstoptime on builds  (cost=0.57..68939.47 rows=243 width=120) (actual time=114.857..114.857 rows=0.00 loops=1)
         Index Cond: (finished = 0)
         Filter: (jobset_id = 123)
         Rows Removed by Filter: 196472
         Index Searches: 1
         Buffers: shared hit=78753
 Planning:
   Buffers: shared hit=324
 Planning Time: 1.515 ms
 Execution Time: 114.872 ms
13:00:01
@hexa:lossy.networkhexa I think an index that includes the jobset could help 13:01:07
@hexa:lossy.networkhexaRedacted or Malformed Event13:01:38
@hexa:lossy.networkhexa
                                                                           QUERY PLAN                                                                           
----------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.57..51.01 rows=50 width=120) (actual time=0.306..0.372 rows=50.00 loops=1)
   Buffers: shared hit=53 read=1
   ->  Index Scan using builds_jobset_finished_stoptime_idx on builds  (cost=0.57..27046.02 rows=26812 width=120) (actual time=0.306..0.369 rows=50.00 loops=1)
         Index Cond: ((jobset_id = 318) AND (finished = 0))
         Index Searches: 1
         Buffers: shared hit=53 read=1
 Planning:
   Buffers: shared hit=16
 Planning Time: 0.476 ms
 Execution Time: 0.380 ms
13:13:09
@hexa:lossy.networkhexayeah, looks like the query backlog is now gone13:14:26
@vcunat:matrix.orgVladimír ČunátThe evaluator isn't able to survive a few seconds of DB outage. It was dead and had to be restarted manually.15:30:07
@joerg:thalheim.ioMic92 hexa (signing key rotation when): so are we hammering the database with our scheduler now too much? 17:00:39
@joerg:thalheim.ioMic92I finally got some time to do look at stuff17:00:52
@hexa:lossy.networkhexaI added a new index and it was fine17:01:39
@hexa:lossy.networkhexa
CREATE INDEX CONCURRENTLY builds_jobset_finished_stoptime_idx
ON builds (jobset_id, finished, stoptime DESC);
17:01:56
@hexa:lossy.networkhexaRedacted or Malformed Event17:01:59
@hexa:lossy.networkhexafwiw17:02:00
@hexa:lossy.networkhexathe cost is ~5.6 GiB17:02:35
@k900:0upti.meK900 We still haven't figured out partitioning on builds, have we? 17:06:03
@hexa:lossy.networkhexawe have not17:06:17
@hexa:lossy.networkhexait would the good long-term move17:06:28
@hexa:lossy.networkhexabesides pruning the database from old builds17:06:36
@hexa:lossy.networkhexaRedacted or Malformed Event17:06:40
@joerg:thalheim.ioMic92and that is blocked on doing gc.17:07:49
@joerg:thalheim.ioMic92looks like deadlocks are still happening17:15:54
@joerg:thalheim.ioMic92it's more of a write issue17:16:03
@joerg:thalheim.ioMic92 vcunat: looks like queue-runner and hydra-evaluator are competing over the same rows 17:16:50
@joerg:thalheim.ioMic92Sounds plausible?17:16:55
@joerg:thalheim.ioMic92So I think we can't really avoid this serialization conflict but we can make the two services tolerate and retry17:25:26

Show newer messages


Back to Room ListRoom Version: 6