| 9 Feb 2022 |
 Amanda (she/her) | To be specific, this is what I start to see, though it's not always hydra-gc-update-roots:
MESSAGE DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=hydra;user=hydra;','',...) failed: FATAL: remaining connection slots are reserved for non-replication superuser connections at /nix/store/12d5ignyfdq5kpdj2hk5yympd9rdkw49-hydra-perl-deps/lib/perl5/site_perl/5.34.0/DBIx/Class/Storage/DBI.pm line 1639. at /nix/store/5bqzhwky888dyfyv133ak4nzqfaa98bx-hydra-2021-08-11/bin/.hydra-update-gc-roots-wrapped line 72
| 00:17:23 |
 das_j | ah yeah looks like we have max_connections = 250; set | 00:17:42 |
| Amanda (she/her) | I think the last time this crept up on me it was hydra-queue-runner which started falling over due to it | 00:18:34 |
 @grahamc:nixos.org | Amanda (she/her): can you list open connections? they should be identified which "app" they're coming from | 00:33:00 |
 ajs124 | In reply to @janne.hess:helsinki-systems.de
Maybe we are not running into it because we are not running postgres with default settings… we used to, before we adjusted the settings. remember the nice error message catalyst always throws up? the one that's translated to austrian, for some reason? | 00:36:58 |
| @grahamc:nixos.org | lmao | 00:37:12 |
| ajs124 | 
Download image.png | 00:37:49 |
| ajs124 | that's our hydra host. don't think that db is used for anything else. | 00:38:08 |
| @grahamc:nixos.org | it may be the webserver holding open connections one in each worker | 00:38:27 |
| @grahamc:nixos.org | it'd be interesting for you to list the app of each open connection :) | 00:38:49 |
| @grahamc:nixos.org | ("app" is a feature of postgres) | 00:38:58 |
| ajs124 | is that the application_name in pg_stat_activity or something else? | 00:40:25 |
| @grahamc:nixos.org | ah yeah that's what I'm thinking | 00:40:32 |
| ajs124 | looks like hydra doesn't set that, then | 00:40:52 |
| @grahamc:nixos.org | it, uh, should | 00:40:57 |
| @grahamc:nixos.org | https://github.com/NixOS/nixpkgs/blob/3ecddf791da4d893beb35fb09eb9da55b326f4fb/nixos/modules/services/continuous-integration/hydra/default.nix#L93 | 00:41:10 |
| ajs124 | process_id | username | database_name | client_address | application_name | backend_start | state | state_change
------------+----------+---------------+----------------+------------------+-------------------------------+--------+-------------------------------
767789 | | | | | 2022-01-31 03:21:20.761112+00 | |
767791 | postgres | | | | 2022-01-31 03:21:20.761672+00 | |
2100745 | hydra | hydra | | | 2022-02-06 03:21:51.972479+00 | idle | 2022-02-06 03:21:51.976446+00
2100749 | hydra | hydra | | | 2022-02-06 03:21:51.976969+00 | idle | 2022-02-09 00:40:25.835551+00
2100750 | hydra | hydra | | | 2022-02-06 03:21:51.977956+00 | idle | 2022-02-09 00:07:40.605865+00
2100757 | hydra | hydra | | | 2022-02-06 03:21:51.982028+00 | idle | 2022-02-09 00:40:32.782287+00
4122292 | hydra | hydra | | | 2022-02-09 00:03:06.790509+00 | idle | 2022-02-09 00:40:15.03025+00
2193464 | hydra | hydra | | | 2022-02-06 03:22:47.258413+00 | idle | 2022-02-08 21:08:13.317336+00
3744105 | hydra | hydra | | | 2022-02-07 21:22:11.388513+00 | idle | 2022-02-09 00:24:38.972683+00
2100767 | hydra | hydra | | | 2022-02-06 03:21:53.419122+00 | idle | 2022-02-09 00:24:50.845463+00
154217 | hydra | hydra | | | 2022-02-07 21:43:57.385992+00 | idle | 2022-02-07 21:44:15.032245+00
2920004 | hydra | hydra | | | 2022-02-06 15:47:00.407801+00 | idle | 2022-02-08 16:49:09.376064+00
2697292 | hydra | hydra | | | 2022-02-06 10:32:09.713231+00 | idle | 2022-02-07 21:43:59.564759+00
2193497 | hydra | hydra | | | 2022-02-06 03:22:48.727351+00 | idle | 2022-02-07 21:44:15.718113+00
2193500 | hydra | hydra | | | 2022-02-06 03:22:49.884918+00 | idle | 2022-02-08 16:49:09.865934+00
2193503 | hydra | hydra | | | 2022-02-06 03:22:50.003524+00 | idle | 2022-02-09 00:11:01.617221+00
2193506 | hydra | hydra | | | 2022-02-06 03:22:50.256277+00 | idle | 2022-02-08 16:49:10.062656+00
772478 | postgres | postgres | | | 2022-01-31 03:23:11.566635+00 | idle | 2022-02-09 00:40:12.334692+00
2193672 | hydra | hydra | | | 2022-02-06 03:23:02.67438+00 | idle | 2022-02-09 00:07:59.696627+00
2193673 | hydra | hydra | | | 2022-02-06 03:23:02.674803+00 | idle | 2022-02-09 00:07:30.005679+00
2144989 | hydra | hydra | | | 2022-02-06 04:31:44.85043+00 | idle | 2022-02-09 00:07:47.918416+00
2194033 | hydra | hydra | | | 2022-02-06 03:23:14.463385+00 | idle | 2022-02-08 17:27:23.360953+00
2194147 | hydra | hydra | | | 2022-02-06 03:23:15.504526+00 | idle | 2022-02-09 00:14:33.576331+00
134419 | hydra | hydra | | | 2022-02-06 04:24:43.61403+00 | idle | 2022-02-08 21:07:17.589039+00
3744112 | hydra | hydra | | | 2022-02-07 21:22:11.389023+00 | idle | 2022-02-07 21:44:14.480124+00
16623 | hydra | hydra | | | 2022-02-06 11:08:36.826006+00 | idle | 2022-02-08 16:49:09.756905+00
2920005 | hydra | hydra | | | 2022-02-06 15:47:00.408248+00 | idle | 2022-02-09 00:08:09.465044+00
2920007 | hydra | hydra | | | 2022-02-06 15:47:00.408773+00 | idle | 2022-02-08 16:49:09.434839+00
4122296 | hydra | hydra | | | 2022-02-09 00:03:07.202732+00 | idle | 2022-02-09 00:40:25.042181+00
16626 | hydra | hydra | | | 2022-02-06 11:08:36.826591+00 | idle | 2022-02-07 21:44:15.686944+00
16628 | hydra | hydra | | | 2022-02-06 11:08:36.827126+00 | idle | 2022-02-08 16:49:09.798568+00
16629 | hydra | hydra | | | 2022-02-06 11:08:36.827646+00 | idle | 2022-02-09 00:08:06.127498+00
16633 | hydra | hydra | | | 2022-02-06 11:08:36.828381+00 | idle | 2022-02-08 21:06:45.503025+00
16635 | hydra | hydra | | | 2022-02-06 11:08:36.82892+00 | idle | 2022-02-07 21:44:19.029125+00
16636 | hydra | hydra | | | 2022-02-06 11:08:36.829488+00 | idle | 2022-02-08 16:49:12.609133+00
16637 | hydra | hydra | | | 2022-02-06 11:08:36.830016+00 | idle | 2022-02-08 16:49:10.07487+00
16638 | hydra | hydra | | | 2022-02-06 11:08:36.830835+00 | idle | 2022-02-08 16:49:09.735638+00
16643 | hydra | hydra | | | 2022-02-06 11:08:36.83178+00 | idle | 2022-02-07 21:44:16.327929+00
16646 | hydra | hydra | | | 2022-02-06 11:08:36.832366+00 | idle | 2022-02-08 16:49:10.116202+00
16647 | hydra | hydra | | | 2022-02-06 11:08:36.832995+00 | idle | 2022-02-09 00:07:46.520188+00
16648 | hydra | hydra | | | 2022-02-06 11:08:36.833501+00 | idle | 2022-02-07 21:47:35.948999+00
16651 | hydra | hydra | | | 2022-02-06 11:08:36.841643+00 | idle | 2022-02-08 16:49:10.223156+00
2920008 | hydra | hydra | | | 2022-02-06 15:47:00.40929+00 | idle | 2022-02-08 16:49:10.146286+00
2920009 | hydra | hydra | | | 2022-02-06 15:47:00.409851+00 | idle | 2022-02-08 21:07:17.60219+00
2920003 | hydra | hydra | | | 2022-02-06 15:47:00.407224+00 | idle | 2022-02-08 16:49:12.512929+00
2920219 | hydra | hydra | | | 2022-02-06 15:47:00.575394+00 | idle | 2022-02-08 16:49:09.452918+00
1957230 | hydra | hydra | | | 2022-02-09 00:15:33.157612+00 | idle | 2022-02-09 00:40:35.05359+00
3744115 | hydra | hydra | | | 2022-02-07 21:22:11.389461+00 | idle | 2022-02-09 00:07:46.441919+00
1328140 | hydra | hydra | | | 2022-02-06 16:20:48.531341+00 | idle | 2022-02-08 16:49:10.146765+00
1328173 | hydra | hydra | | | 2022-02-06 16:20:48.550221+00 | idle | 2022-02-09 00:07:51.623894+00
4113664 | hydra | hydra | | | 2022-02-08 23:59:14.939199+00 | idle | 2022-02-09 00:40:05.017622+00
3744116 | hydra | hydra | | | 2022-02-07 21:22:11.389827+00 | idle | 2022-02-07 21:44:21.485721+00
3744117 | hydra | hydra | | | 2022-02-07 21:22:11.390284+00 | idle | 2022-02-08 16:48:58.993997+00
3744118 | hydra | hydra | | | 2022-02-07 21:22:11.390725+00 | idle | 2022-02-08 16:48:59.561921+00
3744119 | hydra | hydra | | | 2022-02-07 21:22:11.391315+00 | idle | 2022-02-09 00:16:05.659495+00
3744120 | hydra | hydra | | | 2022-02-07 21:22:11.391845+00 | idle | 2022-02-07 21:43:59.033467+00
3744121 | hydra | hydra | | | 2022-02-07 21:22:11.392368+00 | idle | 2022-02-08 16:49:09.97798+00
3744122 | hydra | hydra | | | 2022-02-07 21:22:11.392948+00 | idle | 2022-02-07 21:44:20.381324+00
3744123 | hydra | hydra | | | 2022-02-07 21:22:11.393491+00 | idle | 2022-02-08 16:49:09.418377+00
3744124 | hydra | hydra | | | 2022-02-07 21:22:11.394069+00 | idle | 2022-02-08 17:27:23.377257+00
3744125 | hydra | hydra | | | 2022-02-07 21:22:11.394579+00 | idle | 2022-02-07 21:44:00.441137+00
3744126 | hydra | hydra | | | 2022-02-07 21:22:11.395094+00 | idle | 2022-02-09 00:08:33.296508+00
3744127 | hydra | hydra | | | 2022-02-07 21:22:11.395692+00 | idle | 2022-02-08 16:49:09.08391+00
3744128 | hydra | hydra | | | 2022-02-07 21:22:11.396204+00 | idle | 2022-02-09 00:07:30.474033+00
3744129 | hydra | hydra | | | 2022-02-07 21:22:11.396775+00 | idle | 2022-02-07 21:44:10.400999+00
3744130 | hydra | hydra | | | 2022-02-07 21:22:11.397558+00 | idle | 2022-02-08 21:07:35.019719+00
3744131 | hydra | hydra | | | 2022-02-07 21:22:11.39868+00 | idle | 2022-02-09 00:07:41.046443+00
3744133 | hydra | hydra | | | 2022-02-07 21:22:11.400223+00 | idle | 2022-02-09 00:07:48.000959+00
3745428 | hydra | hydra | | | 2022-02-07 21:22:13.511961+00 | idle | 2022-02-08 16:49:09.685652+00
3748917 | hydra | hydra | | | 2022-02-07 21:22:21.978876+00 | idle | 2022-02-09 00:08:50.208568+00
3749186 | hydra | hydra | | | 2022-02-07 21:22:22.794779+00 | idle | 2022-02-09 00:07:40.676103+00
3749222 | hydra | hydra | | | 2022-02-07 21:22:22.879091+00 | idle | 2022-02-08 16:49:09.800782+00
3768185 | hydra | hydra | | | 2022-02-07 21:22:48.089565+00 | idle | 2022-02-07 21:44:03.913339+00
3768561 | hydra | hydra | | | 2022-02-07 21:22:49.057463+00 | idle | 2022-02-08 17:27:23.600636+00
| 00:41:10 |
| @grahamc:nixos.org | https://github.com/NixOS/hydra/blob/3864ca820adba64fa4d300e98994db9ffc44bd06/hydra-module.nix#L62 | 00:41:14 |
| ajs124 | so either I'm holding it wrong (likely) or the documentation doesn't match the code (also likely) | 00:43:19 |
| @grahamc:nixos.org | can you check the HYDRA_DBI env var for the various services? | 00:43:42 |
| @grahamc:nixos.org | or (even better) in the /proc/.../env file | 00:43:53 |
| ajs124 | I took to random pids (from the list above) and /proc/…/environ was empty for them | 00:44:59 |
| @grahamc:nixos.org | ah they probably clear the env like they should | 00:45:17 |
| @grahamc:nixos.org | anyway, does the service file include the HYDRA_DBI var? | 00:50:56 |
| ajs124 | yeah, that seems to be there | 00:52:50 |
| @grahamc:nixos.org | and does it have the application name? | 00:52:59 |
| ajs124 |
Environment="HYDRA_DBI=dbi:Pg:user=hydra;dbname=hydra;;application_name=hydra-queue-runner"
Environment="HYDRA_DBI=dbi:Pg:user=hydra;dbname=hydra;;application_name=hydra-notify"
| 00:53:15 |
| @grahamc:nixos.org | 🤔 | 00:53:28 |
| @grahamc:nixos.org | 🤷 | 00:53:56 |
 hexa | In reply to @grahamc:nixos.org
🤷 🤷♀️ | 01:09:52 |
