| 9 Feb 2022 |
 das_j | ah yeah looks like we have max_connections = 250; set | 00:17:42 |
 Amanda (she/her) | I think the last time this crept up on me it was hydra-queue-runner which started falling over due to it | 00:18:34 |
 @grahamc:nixos.org | Amanda (she/her): can you list open connections? they should be identified which "app" they're coming from | 00:33:00 |
 ajs124 | In reply to @janne.hess:helsinki-systems.de
Maybe we are not running into it because we are not running postgres with default settings… we used to, before we adjusted the settings. remember the nice error message catalyst always throws up? the one that's translated to austrian, for some reason? | 00:36:58 |
| @grahamc:nixos.org | lmao | 00:37:12 |
| ajs124 | 
Download image.png | 00:37:49 |
| ajs124 | that's our hydra host. don't think that db is used for anything else. | 00:38:08 |
| @grahamc:nixos.org | it may be the webserver holding open connections one in each worker | 00:38:27 |
| @grahamc:nixos.org | it'd be interesting for you to list the app of each open connection :) | 00:38:49 |
| @grahamc:nixos.org | ("app" is a feature of postgres) | 00:38:58 |
| ajs124 | is that the application_name in pg_stat_activity or something else? | 00:40:25 |
| @grahamc:nixos.org | ah yeah that's what I'm thinking | 00:40:32 |
| ajs124 | looks like hydra doesn't set that, then | 00:40:52 |
| @grahamc:nixos.org | it, uh, should | 00:40:57 |
| @grahamc:nixos.org | https://github.com/NixOS/nixpkgs/blob/3ecddf791da4d893beb35fb09eb9da55b326f4fb/nixos/modules/services/continuous-integration/hydra/default.nix#L93 | 00:41:10 |
| ajs124 | process_id | username | database_name | client_address | application_name | backend_start | state | state_change
------------+----------+---------------+----------------+------------------+-------------------------------+--------+-------------------------------
767789 | | | | | 2022-01-31 03:21:20.761112+00 | |
767791 | postgres | | | | 2022-01-31 03:21:20.761672+00 | |
2100745 | hydra | hydra | | | 2022-02-06 03:21:51.972479+00 | idle | 2022-02-06 03:21:51.976446+00
2100749 | hydra | hydra | | | 2022-02-06 03:21:51.976969+00 | idle | 2022-02-09 00:40:25.835551+00
2100750 | hydra | hydra | | | 2022-02-06 03:21:51.977956+00 | idle | 2022-02-09 00:07:40.605865+00
2100757 | hydra | hydra | | | 2022-02-06 03:21:51.982028+00 | idle | 2022-02-09 00:40:32.782287+00
4122292 | hydra | hydra | | | 2022-02-09 00:03:06.790509+00 | idle | 2022-02-09 00:40:15.03025+00
2193464 | hydra | hydra | | | 2022-02-06 03:22:47.258413+00 | idle | 2022-02-08 21:08:13.317336+00
3744105 | hydra | hydra | | | 2022-02-07 21:22:11.388513+00 | idle | 2022-02-09 00:24:38.972683+00
2100767 | hydra | hydra | | | 2022-02-06 03:21:53.419122+00 | idle | 2022-02-09 00:24:50.845463+00
154217 | hydra | hydra | | | 2022-02-07 21:43:57.385992+00 | idle | 2022-02-07 21:44:15.032245+00
2920004 | hydra | hydra | | | 2022-02-06 15:47:00.407801+00 | idle | 2022-02-08 16:49:09.376064+00
2697292 | hydra | hydra | | | 2022-02-06 10:32:09.713231+00 | idle | 2022-02-07 21:43:59.564759+00
2193497 | hydra | hydra | | | 2022-02-06 03:22:48.727351+00 | idle | 2022-02-07 21:44:15.718113+00
2193500 | hydra | hydra | | | 2022-02-06 03:22:49.884918+00 | idle | 2022-02-08 16:49:09.865934+00
2193503 | hydra | hydra | | | 2022-02-06 03:22:50.003524+00 | idle | 2022-02-09 00:11:01.617221+00
2193506 | hydra | hydra | | | 2022-02-06 03:22:50.256277+00 | idle | 2022-02-08 16:49:10.062656+00
772478 | postgres | postgres | | | 2022-01-31 03:23:11.566635+00 | idle | 2022-02-09 00:40:12.334692+00
2193672 | hydra | hydra | | | 2022-02-06 03:23:02.67438+00 | idle | 2022-02-09 00:07:59.696627+00
2193673 | hydra | hydra | | | 2022-02-06 03:23:02.674803+00 | idle | 2022-02-09 00:07:30.005679+00
2144989 | hydra | hydra | | | 2022-02-06 04:31:44.85043+00 | idle | 2022-02-09 00:07:47.918416+00
2194033 | hydra | hydra | | | 2022-02-06 03:23:14.463385+00 | idle | 2022-02-08 17:27:23.360953+00
2194147 | hydra | hydra | | | 2022-02-06 03:23:15.504526+00 | idle | 2022-02-09 00:14:33.576331+00
134419 | hydra | hydra | | | 2022-02-06 04:24:43.61403+00 | idle | 2022-02-08 21:07:17.589039+00
3744112 | hydra | hydra | | | 2022-02-07 21:22:11.389023+00 | idle | 2022-02-07 21:44:14.480124+00
16623 | hydra | hydra | | | 2022-02-06 11:08:36.826006+00 | idle | 2022-02-08 16:49:09.756905+00
2920005 | hydra | hydra | | | 2022-02-06 15:47:00.408248+00 | idle | 2022-02-09 00:08:09.465044+00
2920007 | hydra | hydra | | | 2022-02-06 15:47:00.408773+00 | idle | 2022-02-08 16:49:09.434839+00
4122296 | hydra | hydra | | | 2022-02-09 00:03:07.202732+00 | idle | 2022-02-09 00:40:25.042181+00
16626 | hydra | hydra | | | 2022-02-06 11:08:36.826591+00 | idle | 2022-02-07 21:44:15.686944+00
16628 | hydra | hydra | | | 2022-02-06 11:08:36.827126+00 | idle | 2022-02-08 16:49:09.798568+00
16629 | hydra | hydra | | | 2022-02-06 11:08:36.827646+00 | idle | 2022-02-09 00:08:06.127498+00
16633 | hydra | hydra | | | 2022-02-06 11:08:36.828381+00 | idle | 2022-02-08 21:06:45.503025+00
16635 | hydra | hydra | | | 2022-02-06 11:08:36.82892+00 | idle | 2022-02-07 21:44:19.029125+00
16636 | hydra | hydra | | | 2022-02-06 11:08:36.829488+00 | idle | 2022-02-08 16:49:12.609133+00
16637 | hydra | hydra | | | 2022-02-06 11:08:36.830016+00 | idle | 2022-02-08 16:49:10.07487+00
16638 | hydra | hydra | | | 2022-02-06 11:08:36.830835+00 | idle | 2022-02-08 16:49:09.735638+00
16643 | hydra | hydra | | | 2022-02-06 11:08:36.83178+00 | idle | 2022-02-07 21:44:16.327929+00
16646 | hydra | hydra | | | 2022-02-06 11:08:36.832366+00 | idle | 2022-02-08 16:49:10.116202+00
16647 | hydra | hydra | | | 2022-02-06 11:08:36.832995+00 | idle | 2022-02-09 00:07:46.520188+00
16648 | hydra | hydra | | | 2022-02-06 11:08:36.833501+00 | idle | 2022-02-07 21:47:35.948999+00
16651 | hydra | hydra | | | 2022-02-06 11:08:36.841643+00 | idle | 2022-02-08 16:49:10.223156+00
2920008 | hydra | hydra | | | 2022-02-06 15:47:00.40929+00 | idle | 2022-02-08 16:49:10.146286+00
2920009 | hydra | hydra | | | 2022-02-06 15:47:00.409851+00 | idle | 2022-02-08 21:07:17.60219+00
2920003 | hydra | hydra | | | 2022-02-06 15:47:00.407224+00 | idle | 2022-02-08 16:49:12.512929+00
2920219 | hydra | hydra | | | 2022-02-06 15:47:00.575394+00 | idle | 2022-02-08 16:49:09.452918+00
1957230 | hydra | hydra | | | 2022-02-09 00:15:33.157612+00 | idle | 2022-02-09 00:40:35.05359+00
3744115 | hydra | hydra | | | 2022-02-07 21:22:11.389461+00 | idle | 2022-02-09 00:07:46.441919+00
1328140 | hydra | hydra | | | 2022-02-06 16:20:48.531341+00 | idle | 2022-02-08 16:49:10.146765+00
1328173 | hydra | hydra | | | 2022-02-06 16:20:48.550221+00 | idle | 2022-02-09 00:07:51.623894+00
4113664 | hydra | hydra | | | 2022-02-08 23:59:14.939199+00 | idle | 2022-02-09 00:40:05.017622+00
3744116 | hydra | hydra | | | 2022-02-07 21:22:11.389827+00 | idle | 2022-02-07 21:44:21.485721+00
3744117 | hydra | hydra | | | 2022-02-07 21:22:11.390284+00 | idle | 2022-02-08 16:48:58.993997+00
3744118 | hydra | hydra | | | 2022-02-07 21:22:11.390725+00 | idle | 2022-02-08 16:48:59.561921+00
3744119 | hydra | hydra | | | 2022-02-07 21:22:11.391315+00 | idle | 2022-02-09 00:16:05.659495+00
3744120 | hydra | hydra | | | 2022-02-07 21:22:11.391845+00 | idle | 2022-02-07 21:43:59.033467+00
3744121 | hydra | hydra | | | 2022-02-07 21:22:11.392368+00 | idle | 2022-02-08 16:49:09.97798+00
3744122 | hydra | hydra | | | 2022-02-07 21:22:11.392948+00 | idle | 2022-02-07 21:44:20.381324+00
3744123 | hydra | hydra | | | 2022-02-07 21:22:11.393491+00 | idle | 2022-02-08 16:49:09.418377+00
3744124 | hydra | hydra | | | 2022-02-07 21:22:11.394069+00 | idle | 2022-02-08 17:27:23.377257+00
3744125 | hydra | hydra | | | 2022-02-07 21:22:11.394579+00 | idle | 2022-02-07 21:44:00.441137+00
3744126 | hydra | hydra | | | 2022-02-07 21:22:11.395094+00 | idle | 2022-02-09 00:08:33.296508+00
3744127 | hydra | hydra | | | 2022-02-07 21:22:11.395692+00 | idle | 2022-02-08 16:49:09.08391+00
3744128 | hydra | hydra | | | 2022-02-07 21:22:11.396204+00 | idle | 2022-02-09 00:07:30.474033+00
3744129 | hydra | hydra | | | 2022-02-07 21:22:11.396775+00 | idle | 2022-02-07 21:44:10.400999+00
3744130 | hydra | hydra | | | 2022-02-07 21:22:11.397558+00 | idle | 2022-02-08 21:07:35.019719+00
3744131 | hydra | hydra | | | 2022-02-07 21:22:11.39868+00 | idle | 2022-02-09 00:07:41.046443+00
3744133 | hydra | hydra | | | 2022-02-07 21:22:11.400223+00 | idle | 2022-02-09 00:07:48.000959+00
3745428 | hydra | hydra | | | 2022-02-07 21:22:13.511961+00 | idle | 2022-02-08 16:49:09.685652+00
3748917 | hydra | hydra | | | 2022-02-07 21:22:21.978876+00 | idle | 2022-02-09 00:08:50.208568+00
3749186 | hydra | hydra | | | 2022-02-07 21:22:22.794779+00 | idle | 2022-02-09 00:07:40.676103+00
3749222 | hydra | hydra | | | 2022-02-07 21:22:22.879091+00 | idle | 2022-02-08 16:49:09.800782+00
3768185 | hydra | hydra | | | 2022-02-07 21:22:48.089565+00 | idle | 2022-02-07 21:44:03.913339+00
3768561 | hydra | hydra | | | 2022-02-07 21:22:49.057463+00 | idle | 2022-02-08 17:27:23.600636+00
| 00:41:10 |
| @grahamc:nixos.org | https://github.com/NixOS/hydra/blob/3864ca820adba64fa4d300e98994db9ffc44bd06/hydra-module.nix#L62 | 00:41:14 |
| ajs124 | so either I'm holding it wrong (likely) or the documentation doesn't match the code (also likely) | 00:43:19 |
| @grahamc:nixos.org | can you check the HYDRA_DBI env var for the various services? | 00:43:42 |
| @grahamc:nixos.org | or (even better) in the /proc/.../env file | 00:43:53 |
| ajs124 | I took to random pids (from the list above) and /proc/…/environ was empty for them | 00:44:59 |
| @grahamc:nixos.org | ah they probably clear the env like they should | 00:45:17 |
| @grahamc:nixos.org | anyway, does the service file include the HYDRA_DBI var? | 00:50:56 |
| ajs124 | yeah, that seems to be there | 00:52:50 |
| @grahamc:nixos.org | and does it have the application name? | 00:52:59 |
| ajs124 |
Environment="HYDRA_DBI=dbi:Pg:user=hydra;dbname=hydra;;application_name=hydra-queue-runner"
Environment="HYDRA_DBI=dbi:Pg:user=hydra;dbname=hydra;;application_name=hydra-notify"
| 00:53:15 |
| @grahamc:nixos.org | 🤔 | 00:53:28 |
| @grahamc:nixos.org | 🤷 | 00:53:56 |
 hexa | In reply to @grahamc:nixos.org
🤷 🤷♀️ | 01:09:52 |
| Amanda (she/her) | In reply to @grahamc:nixos.org
Amanda (she/her): can you list open connections? they should be identified which "app" they're coming from Yeah, I tried that when it first happen then I forgot to report that the app name wasn't showing up fr whatever reason, then forgot about the problem enitely until it happen again just now and my alerts channel was crying about the hydra gc service failing | 02:58:03 |
