| 8 Feb 2022 |
 das_j | why even bother with releases then? | 15:33:55 |
 @grahamc:nixos.org | =) | 15:37:32 |
 ajs124 | follow the minio model. release every Friday (I think). | 15:53:33 |
 hexa | let fools deploy over the weekend | 16:05:03 |
| hexa | * let fools crash and burn over the weekend | 16:05:12 |
| @grahamc:nixos.org | there you go | 17:06:01 |
| @grahamc:nixos.org | https://discourse.nixos.org/t/dropping-jobsetinputalts/17584 | 17:55:16 |
| 9 Feb 2022 |
 Amanda (she/her) | Anybody else have hydra seem to slowly be leaking postgres handles? It seems to slowly build up, and I'm not even sure which service it is that it's happening from | 00:14:41 |
| Amanda (she/her) | Eventually stuff starts falling over because postgres is like "No. Stop." | 00:15:02 |
| das_j | Maybe we are not running into it because we are not running postgres with default settings… | 00:16:33 |
| Amanda (she/her) | To be specific, this is what I start to see, though it's not always hydra-gc-update-roots:
MESSAGE DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=hydra;user=hydra;','',...) failed: FATAL: remaining connection slots are reserved for non-replication superuser connections at /nix/store/12d5ignyfdq5kpdj2hk5yympd9rdkw49-hydra-perl-deps/lib/perl5/site_perl/5.34.0/DBIx/Class/Storage/DBI.pm line 1639. at /nix/store/5bqzhwky888dyfyv133ak4nzqfaa98bx-hydra-2021-08-11/bin/.hydra-update-gc-roots-wrapped line 72
| 00:17:23 |
| das_j | ah yeah looks like we have max_connections = 250; set | 00:17:42 |
| Amanda (she/her) | I think the last time this crept up on me it was hydra-queue-runner which started falling over due to it | 00:18:34 |
| @grahamc:nixos.org | Amanda (she/her): can you list open connections? they should be identified which "app" they're coming from | 00:33:00 |
| ajs124 | In reply to @janne.hess:helsinki-systems.de
Maybe we are not running into it because we are not running postgres with default settings… we used to, before we adjusted the settings. remember the nice error message catalyst always throws up? the one that's translated to austrian, for some reason? | 00:36:58 |
| @grahamc:nixos.org | lmao | 00:37:12 |
| ajs124 | 
Download image.png | 00:37:49 |
| ajs124 | that's our hydra host. don't think that db is used for anything else. | 00:38:08 |
| @grahamc:nixos.org | it may be the webserver holding open connections one in each worker | 00:38:27 |
| @grahamc:nixos.org | it'd be interesting for you to list the app of each open connection :) | 00:38:49 |
| @grahamc:nixos.org | ("app" is a feature of postgres) | 00:38:58 |
| ajs124 | is that the application_name in pg_stat_activity or something else? | 00:40:25 |
| @grahamc:nixos.org | ah yeah that's what I'm thinking | 00:40:32 |
| ajs124 | looks like hydra doesn't set that, then | 00:40:52 |
| @grahamc:nixos.org | it, uh, should | 00:40:57 |
| @grahamc:nixos.org | https://github.com/NixOS/nixpkgs/blob/3ecddf791da4d893beb35fb09eb9da55b326f4fb/nixos/modules/services/continuous-integration/hydra/default.nix#L93 | 00:41:10 |
| ajs124 | process_id | username | database_name | client_address | application_name | backend_start | state | state_change
------------+----------+---------------+----------------+------------------+-------------------------------+--------+-------------------------------
767789 | | | | | 2022-01-31 03:21:20.761112+00 | |
767791 | postgres | | | | 2022-01-31 03:21:20.761672+00 | |
2100745 | hydra | hydra | | | 2022-02-06 03:21:51.972479+00 | idle | 2022-02-06 03:21:51.976446+00
2100749 | hydra | hydra | | | 2022-02-06 03:21:51.976969+00 | idle | 2022-02-09 00:40:25.835551+00
2100750 | hydra | hydra | | | 2022-02-06 03:21:51.977956+00 | idle | 2022-02-09 00:07:40.605865+00
2100757 | hydra | hydra | | | 2022-02-06 03:21:51.982028+00 | idle | 2022-02-09 00:40:32.782287+00
4122292 | hydra | hydra | | | 2022-02-09 00:03:06.790509+00 | idle | 2022-02-09 00:40:15.03025+00
2193464 | hydra | hydra | | | 2022-02-06 03:22:47.258413+00 | idle | 2022-02-08 21:08:13.317336+00
3744105 | hydra | hydra | | | 2022-02-07 21:22:11.388513+00 | idle | 2022-02-09 00:24:38.972683+00
2100767 | hydra | hydra | | | 2022-02-06 03:21:53.419122+00 | idle | 2022-02-09 00:24:50.845463+00
154217 | hydra | hydra | | | 2022-02-07 21:43:57.385992+00 | idle | 2022-02-07 21:44:15.032245+00
2920004 | hydra | hydra | | | 2022-02-06 15:47:00.407801+00 | idle | 2022-02-08 16:49:09.376064+00
2697292 | hydra | hydra | | | 2022-02-06 10:32:09.713231+00 | idle | 2022-02-07 21:43:59.564759+00
2193497 | hydra | hydra | | | 2022-02-06 03:22:48.727351+00 | idle | 2022-02-07 21:44:15.718113+00
2193500 | hydra | hydra | | | 2022-02-06 03:22:49.884918+00 | idle | 2022-02-08 16:49:09.865934+00
2193503 | hydra | hydra | | | 2022-02-06 03:22:50.003524+00 | idle | 2022-02-09 00:11:01.617221+00
2193506 | hydra | hydra | | | 2022-02-06 03:22:50.256277+00 | idle | 2022-02-08 16:49:10.062656+00
772478 | postgres | postgres | | | 2022-01-31 03:23:11.566635+00 | idle | 2022-02-09 00:40:12.334692+00
2193672 | hydra | hydra | | | 2022-02-06 03:23:02.67438+00 | idle | 2022-02-09 00:07:59.696627+00
2193673 | hydra | hydra | | | 2022-02-06 03:23:02.674803+00 | idle | 2022-02-09 00:07:30.005679+00
2144989 | hydra | hydra | | | 2022-02-06 04:31:44.85043+00 | idle | 2022-02-09 00:07:47.918416+00
2194033 | hydra | hydra | | | 2022-02-06 03:23:14.463385+00 | idle | 2022-02-08 17:27:23.360953+00
2194147 | hydra | hydra | | | 2022-02-06 03:23:15.504526+00 | idle | 2022-02-09 00:14:33.576331+00
134419 | hydra | hydra | | | 2022-02-06 04:24:43.61403+00 | idle | 2022-02-08 21:07:17.589039+00
3744112 | hydra | hydra | | | 2022-02-07 21:22:11.389023+00 | idle | 2022-02-07 21:44:14.480124+00
16623 | hydra | hydra | | | 2022-02-06 11:08:36.826006+00 | idle | 2022-02-08 16:49:09.756905+00
2920005 | hydra | hydra | | | 2022-02-06 15:47:00.408248+00 | idle | 2022-02-09 00:08:09.465044+00
2920007 | hydra | hydra | | | 2022-02-06 15:47:00.408773+00 | idle | 2022-02-08 16:49:09.434839+00
4122296 | hydra | hydra | | | 2022-02-09 00:03:07.202732+00 | idle | 2022-02-09 00:40:25.042181+00
16626 | hydra | hydra | | | 2022-02-06 11:08:36.826591+00 | idle | 2022-02-07 21:44:15.686944+00
16628 | hydra | hydra | | | 2022-02-06 11:08:36.827126+00 | idle | 2022-02-08 16:49:09.798568+00
16629 | hydra | hydra | | | 2022-02-06 11:08:36.827646+00 | idle | 2022-02-09 00:08:06.127498+00
16633 | hydra | hydra | | | 2022-02-06 11:08:36.828381+00 | idle | 2022-02-08 21:06:45.503025+00
16635 | hydra | hydra | | | 2022-02-06 11:08:36.82892+00 | idle | 2022-02-07 21:44:19.029125+00
16636 | hydra | hydra | | | 2022-02-06 11:08:36.829488+00 | idle | 2022-02-08 16:49:12.609133+00
16637 | hydra | hydra | | | 2022-02-06 11:08:36.830016+00 | idle | 2022-02-08 16:49:10.07487+00
16638 | hydra | hydra | | | 2022-02-06 11:08:36.830835+00 | idle | 2022-02-08 16:49:09.735638+00
16643 | hydra | hydra | | | 2022-02-06 11:08:36.83178+00 | idle | 2022-02-07 21:44:16.327929+00
16646 | hydra | hydra | | | 2022-02-06 11:08:36.832366+00 | idle | 2022-02-08 16:49:10.116202+00
16647 | hydra | hydra | | | 2022-02-06 11:08:36.832995+00 | idle | 2022-02-09 00:07:46.520188+00
16648 | hydra | hydra | | | 2022-02-06 11:08:36.833501+00 | idle | 2022-02-07 21:47:35.948999+00
16651 | hydra | hydra | | | 2022-02-06 11:08:36.841643+00 | idle | 2022-02-08 16:49:10.223156+00
2920008 | hydra | hydra | | | 2022-02-06 15:47:00.40929+00 | idle | 2022-02-08 16:49:10.146286+00
2920009 | hydra | hydra | | | 2022-02-06 15:47:00.409851+00 | idle | 2022-02-08 21:07:17.60219+00
2920003 | hydra | hydra | | | 2022-02-06 15:47:00.407224+00 | idle | 2022-02-08 16:49:12.512929+00
2920219 | hydra | hydra | | | 2022-02-06 15:47:00.575394+00 | idle | 2022-02-08 16:49:09.452918+00
1957230 | hydra | hydra | | | 2022-02-09 00:15:33.157612+00 | idle | 2022-02-09 00:40:35.05359+00
3744115 | hydra | hydra | | | 2022-02-07 21:22:11.389461+00 | idle | 2022-02-09 00:07:46.441919+00
1328140 | hydra | hydra | | | 2022-02-06 16:20:48.531341+00 | idle | 2022-02-08 16:49:10.146765+00
1328173 | hydra | hydra | | | 2022-02-06 16:20:48.550221+00 | idle | 2022-02-09 00:07:51.623894+00
4113664 | hydra | hydra | | | 2022-02-08 23:59:14.939199+00 | idle | 2022-02-09 00:40:05.017622+00
3744116 | hydra | hydra | | | 2022-02-07 21:22:11.389827+00 | idle | 2022-02-07 21:44:21.485721+00
3744117 | hydra | hydra | | | 2022-02-07 21:22:11.390284+00 | idle | 2022-02-08 16:48:58.993997+00
3744118 | hydra | hydra | | | 2022-02-07 21:22:11.390725+00 | idle | 2022-02-08 16:48:59.561921+00
3744119 | hydra | hydra | | | 2022-02-07 21:22:11.391315+00 | idle | 2022-02-09 00:16:05.659495+00
3744120 | hydra | hydra | | | 2022-02-07 21:22:11.391845+00 | idle | 2022-02-07 21:43:59.033467+00
3744121 | hydra | hydra | | | 2022-02-07 21:22:11.392368+00 | idle | 2022-02-08 16:49:09.97798+00
3744122 | hydra | hydra | | | 2022-02-07 21:22:11.392948+00 | idle | 2022-02-07 21:44:20.381324+00
3744123 | hydra | hydra | | | 2022-02-07 21:22:11.393491+00 | idle | 2022-02-08 16:49:09.418377+00
3744124 | hydra | hydra | | | 2022-02-07 21:22:11.394069+00 | idle | 2022-02-08 17:27:23.377257+00
3744125 | hydra | hydra | | | 2022-02-07 21:22:11.394579+00 | idle | 2022-02-07 21:44:00.441137+00
3744126 | hydra | hydra | | | 2022-02-07 21:22:11.395094+00 | idle | 2022-02-09 00:08:33.296508+00
3744127 | hydra | hydra | | | 2022-02-07 21:22:11.395692+00 | idle | 2022-02-08 16:49:09.08391+00
3744128 | hydra | hydra | | | 2022-02-07 21:22:11.396204+00 | idle | 2022-02-09 00:07:30.474033+00
3744129 | hydra | hydra | | | 2022-02-07 21:22:11.396775+00 | idle | 2022-02-07 21:44:10.400999+00
3744130 | hydra | hydra | | | 2022-02-07 21:22:11.397558+00 | idle | 2022-02-08 21:07:35.019719+00
3744131 | hydra | hydra | | | 2022-02-07 21:22:11.39868+00 | idle | 2022-02-09 00:07:41.046443+00
3744133 | hydra | hydra | | | 2022-02-07 21:22:11.400223+00 | idle | 2022-02-09 00:07:48.000959+00
3745428 | hydra | hydra | | | 2022-02-07 21:22:13.511961+00 | idle | 2022-02-08 16:49:09.685652+00
3748917 | hydra | hydra | | | 2022-02-07 21:22:21.978876+00 | idle | 2022-02-09 00:08:50.208568+00
3749186 | hydra | hydra | | | 2022-02-07 21:22:22.794779+00 | idle | 2022-02-09 00:07:40.676103+00
3749222 | hydra | hydra | | | 2022-02-07 21:22:22.879091+00 | idle | 2022-02-08 16:49:09.800782+00
3768185 | hydra | hydra | | | 2022-02-07 21:22:48.089565+00 | idle | 2022-02-07 21:44:03.913339+00
3768561 | hydra | hydra | | | 2022-02-07 21:22:49.057463+00 | idle | 2022-02-08 17:27:23.600636+00
| 00:41:10 |
| @grahamc:nixos.org | https://github.com/NixOS/hydra/blob/3864ca820adba64fa4d300e98994db9ffc44bd06/hydra-module.nix#L62 | 00:41:14 |
| ajs124 | so either I'm holding it wrong (likely) or the documentation doesn't match the code (also likely) | 00:43:19 |
| @grahamc:nixos.org | can you check the HYDRA_DBI env var for the various services? | 00:43:42 |
