| 9 Feb 2022 |
 @grahamc:nixos.org | ah they probably clear the env like they should | 00:45:17 |
| @grahamc:nixos.org | anyway, does the service file include the HYDRA_DBI var? | 00:50:56 |
 ajs124 | yeah, that seems to be there | 00:52:50 |
| @grahamc:nixos.org | and does it have the application name? | 00:52:59 |
| ajs124 |
Environment="HYDRA_DBI=dbi:Pg:user=hydra;dbname=hydra;;application_name=hydra-queue-runner"
Environment="HYDRA_DBI=dbi:Pg:user=hydra;dbname=hydra;;application_name=hydra-notify"
| 00:53:15 |
| @grahamc:nixos.org | 🤔 | 00:53:28 |
| @grahamc:nixos.org | 🤷 | 00:53:56 |
 hexa | In reply to @grahamc:nixos.org
🤷 🤷♀️ | 01:09:52 |
 Amanda (she/her) | In reply to @grahamc:nixos.org
Amanda (she/her): can you list open connections? they should be identified which "app" they're coming from Yeah, I tried that when it first happen then I forgot to report that the app name wasn't showing up fr whatever reason, then forgot about the problem enitely until it happen again just now and my alerts channel was crying about the hydra gc service failing | 02:58:03 |
| @grahamc:nixos.org | my memory is of this working correctly, sigh, I'll have to test it tomorrow | 02:58:36 |
| Amanda (she/her) | Could it be different in newer postgres? When the old version was removed I leapfrogged to the latest so I don't have to worry about it for awjile | 03:12:16 |
| Amanda (she/her) | ( apologies for the typos, on from my phone and getting ready to sleep soon ) | 03:12:54 |
| ajs124 | In reply to @amanda:camnet.site
Could it be different in newer postgres? When the old version was removed I leapfrogged to the latest so I don't have to worry about it for awjile I think we're on postgres 13
Also, if I'm not misremembering this, nixos-org-configs also raises the connection limit for the hydra.nixos.org | 09:44:10 |
| Amanda (she/her) | Wait, does that double ; throw anything off? Or is that intentional? | 14:41:57 |
| Amanda (she/her) | in the HYDRA_DBI env var | 14:42:11 |
| Amanda (she/her) | Appears not | 14:51:05 |
| Amanda (she/her) | Okay, so | 14:58:40 |
| Amanda (she/her) | Unless the 'COMMIT' psql query differs from the ->commit method, the only place that is causing all these idle connections is hydra-init? | 14:59:24 |
| Amanda (she/her) | oh. It's hydra-queue-runner, I just tried killing active hydra proccesses and seeing which made the entries in pg_stat_activity go away, and it was hydra-queue-runner. For some reason it's got a lot of idle connections with the query of 'COMMIT' | 15:08:08 |
| @grahamc:nixos.org | hmmm | 15:15:44 |
| @grahamc:nixos.org | like the query is still running, or? | 15:15:51 |
| @grahamc:nixos.org | das_j: I'm coming around to test that ldap PR and I am just struggling to get excited about digging in lol | 15:16:19 |
| Amanda (she/her) | grahamc (he/him): Yes, I think that hydra-queue-runner might be leaking the connection. There's 10-20 connections all started at the same time, then never closed | 15:17:27 |
| @grahamc:nixos.org | if the COMMIT is still running then it would not be a leaked connection, it is waiting for postgres to finish | 15:17:50 |
| Amanda (she/her) | pg_stat_actibity has it listed as idle state | 15:18:07 |
| @grahamc:nixos.org | gotcha | 15:18:12 |
| Amanda (she/her) | so I'm working under the assumtion that means commit just was the last query ran | 15:18:27 |
| @grahamc:nixos.org | yeah | 15:18:32 |
| @grahamc:nixos.org | it sounds like it started a worker thread with a connection to handle a build and now it is waiting | 15:18:42 |
| Amanda (she/her) | "At the same time" +- some milliseconds, all within the same couple seconds at least | 15:19:32 |
