| 9 Feb 2022 |
 @grahamc:nixos.org | my memory is of this working correctly, sigh, I'll have to test it tomorrow | 02:58:36 |
 Amanda (she/her) | Could it be different in newer postgres? When the old version was removed I leapfrogged to the latest so I don't have to worry about it for awjile | 03:12:16 |
| Amanda (she/her) | ( apologies for the typos, on from my phone and getting ready to sleep soon ) | 03:12:54 |
 ajs124 | In reply to @amanda:camnet.site
Could it be different in newer postgres? When the old version was removed I leapfrogged to the latest so I don't have to worry about it for awjile I think we're on postgres 13
Also, if I'm not misremembering this, nixos-org-configs also raises the connection limit for the hydra.nixos.org | 09:44:10 |
| Amanda (she/her) | Wait, does that double ; throw anything off? Or is that intentional? | 14:41:57 |
| Amanda (she/her) | in the HYDRA_DBI env var | 14:42:11 |
| Amanda (she/her) | Appears not | 14:51:05 |
| Amanda (she/her) | Okay, so | 14:58:40 |
| Amanda (she/her) | Unless the 'COMMIT' psql query differs from the ->commit method, the only place that is causing all these idle connections is hydra-init? | 14:59:24 |
| Amanda (she/her) | oh. It's hydra-queue-runner, I just tried killing active hydra proccesses and seeing which made the entries in pg_stat_activity go away, and it was hydra-queue-runner. For some reason it's got a lot of idle connections with the query of 'COMMIT' | 15:08:08 |
| @grahamc:nixos.org | hmmm | 15:15:44 |
| @grahamc:nixos.org | like the query is still running, or? | 15:15:51 |
| @grahamc:nixos.org | das_j: I'm coming around to test that ldap PR and I am just struggling to get excited about digging in lol | 15:16:19 |
| Amanda (she/her) | grahamc (he/him): Yes, I think that hydra-queue-runner might be leaking the connection. There's 10-20 connections all started at the same time, then never closed | 15:17:27 |
| @grahamc:nixos.org | if the COMMIT is still running then it would not be a leaked connection, it is waiting for postgres to finish | 15:17:50 |
| Amanda (she/her) | pg_stat_actibity has it listed as idle state | 15:18:07 |
| @grahamc:nixos.org | gotcha | 15:18:12 |
| Amanda (she/her) | so I'm working under the assumtion that means commit just was the last query ran | 15:18:27 |
| @grahamc:nixos.org | yeah | 15:18:32 |
| @grahamc:nixos.org | it sounds like it started a worker thread with a connection to handle a build and now it is waiting | 15:18:42 |
| Amanda (she/her) | "At the same time" +- some milliseconds, all within the same couple seconds at least | 15:19:32 |
| @grahamc:nixos.org | I am hesitant to call it a leak exactly, because the # of connections is very stable for both h.n.o and ajs124 's, but does seem like it would be good to be able to tune that | 15:22:24 |
| Amanda (she/her) | Just checked hydra-queue-runner, lots of messages like this, stedially increasing: warning: 48 concurrent database updates; PostgreSQL may be stalled | 15:24:28 |
| @grahamc:nixos.org | yeah, there's some tuning that would be good | 15:24:52 |
| @grahamc:nixos.org | I wonder if there are some already | 15:24:56 |
| @grahamc:nixos.org | yes! | 15:25:15 |
| @grahamc:nixos.org | max_db_connections defaults to 128 | 15:25:29 |
| Amanda (she/her) | I'm not sure what postgres defaults to, but I've not changed it | 15:28:17 |
| Amanda (she/her) | ( Nor even how to do so ) | 15:29:44 |
| Amanda (she/her) | max_db_connections seems to have been the knob I needed though, danke | 15:32:45 |
