| 9 Feb 2022 |
 Amanda (she/her) | In reply to @grahamc:nixos.org
Amanda (she/her): can you list open connections? they should be identified which "app" they're coming from Yeah, I tried that when it first happen then I forgot to report that the app name wasn't showing up fr whatever reason, then forgot about the problem enitely until it happen again just now and my alerts channel was crying about the hydra gc service failing | 02:58:03 |
 @grahamc:nixos.org | my memory is of this working correctly, sigh, I'll have to test it tomorrow | 02:58:36 |
| Amanda (she/her) | Could it be different in newer postgres? When the old version was removed I leapfrogged to the latest so I don't have to worry about it for awjile | 03:12:16 |
| Amanda (she/her) | ( apologies for the typos, on from my phone and getting ready to sleep soon ) | 03:12:54 |
 ajs124 | In reply to @amanda:camnet.site
Could it be different in newer postgres? When the old version was removed I leapfrogged to the latest so I don't have to worry about it for awjile I think we're on postgres 13
Also, if I'm not misremembering this, nixos-org-configs also raises the connection limit for the hydra.nixos.org | 09:44:10 |
| Amanda (she/her) | Wait, does that double ; throw anything off? Or is that intentional? | 14:41:57 |
| Amanda (she/her) | in the HYDRA_DBI env var | 14:42:11 |
| Amanda (she/her) | Appears not | 14:51:05 |
| Amanda (she/her) | Okay, so | 14:58:40 |
| Amanda (she/her) | Unless the 'COMMIT' psql query differs from the ->commit method, the only place that is causing all these idle connections is hydra-init? | 14:59:24 |
| Amanda (she/her) | oh. It's hydra-queue-runner, I just tried killing active hydra proccesses and seeing which made the entries in pg_stat_activity go away, and it was hydra-queue-runner. For some reason it's got a lot of idle connections with the query of 'COMMIT' | 15:08:08 |
| @grahamc:nixos.org | hmmm | 15:15:44 |
| @grahamc:nixos.org | like the query is still running, or? | 15:15:51 |
| @grahamc:nixos.org | das_j: I'm coming around to test that ldap PR and I am just struggling to get excited about digging in lol | 15:16:19 |
| Amanda (she/her) | grahamc (he/him): Yes, I think that hydra-queue-runner might be leaking the connection. There's 10-20 connections all started at the same time, then never closed | 15:17:27 |
| @grahamc:nixos.org | if the COMMIT is still running then it would not be a leaked connection, it is waiting for postgres to finish | 15:17:50 |
| Amanda (she/her) | pg_stat_actibity has it listed as idle state | 15:18:07 |
| @grahamc:nixos.org | gotcha | 15:18:12 |
| Amanda (she/her) | so I'm working under the assumtion that means commit just was the last query ran | 15:18:27 |
| @grahamc:nixos.org | yeah | 15:18:32 |
| @grahamc:nixos.org | it sounds like it started a worker thread with a connection to handle a build and now it is waiting | 15:18:42 |
| Amanda (she/her) | "At the same time" +- some milliseconds, all within the same couple seconds at least | 15:19:32 |
| @grahamc:nixos.org | I am hesitant to call it a leak exactly, because the # of connections is very stable for both h.n.o and ajs124 's, but does seem like it would be good to be able to tune that | 15:22:24 |
| Amanda (she/her) | Just checked hydra-queue-runner, lots of messages like this, stedially increasing: warning: 48 concurrent database updates; PostgreSQL may be stalled | 15:24:28 |
| @grahamc:nixos.org | yeah, there's some tuning that would be good | 15:24:52 |
| @grahamc:nixos.org | I wonder if there are some already | 15:24:56 |
| @grahamc:nixos.org | yes! | 15:25:15 |
| @grahamc:nixos.org | max_db_connections defaults to 128 | 15:25:29 |
| Amanda (she/her) | I'm not sure what postgres defaults to, but I've not changed it | 15:28:17 |
| Amanda (she/her) | ( Nor even how to do so ) | 15:29:44 |
