| 13 Feb 2022 |
 das_j | yeah that's likely | 17:11:50 |
| das_j | so run 80 again? | 17:12:06 |
 @grahamc:nixos.org | That drops projects? Probably yeah | 17:12:57 |
| das_j | I'll do a backup first :D | 17:13:11 |
| @grahamc:nixos.org | Good idea | 17:15:49 |
| @grahamc:nixos.org | This is, iirc, exactly the reason I mentioned coming up with a better method for tracking migrations :) | 17:16:07 |
| @grahamc:nixos.org | Specifically this migration | 17:16:24 |
| das_j | yess, it works now | 17:19:13 |
| das_j | thanks! | 17:19:14 |
| @grahamc:nixos.org | Great! | 17:26:53 |
| 14 Feb 2022 |
| das_j | I know this solves a solved problem but I wrote a new Hydra exporter in a proper language that can be used for non-official Hydras without patching. Exports all metrics from the queue runner status + reexports the notify metrics | 12:51:24 |
| das_j | Sadly not really compatible with the python exporter because the metrics are now properly namespaced and a bit more consistent :/ | 12:51:46 |
|  @test:boba.best changed their display name from Tseb to Tseb (Old). | 12:52:11 |
 hexa | NIH - not invented at helsinki systems! | 12:53:55 |
| hexa | * NIH - not invented at helsinki systems! 😜 | 12:54:18 |
| das_j | Kinda, yeah :D | 12:54:35 |
| das_j | Also if the code looks like I really wanted to learn some go reflectino - that's what the project was initially for :D | 12:55:02 |
| das_j | * Also if the code looks like I really wanted to learn some go reflectino - that's what the project was initially for | 12:55:04 |
| das_j | * Also if the code looks like I really wanted to learn some go reflection - that's what the project was initially for | 12:55:09 |
| hexa | this looks much more complete from what I remember when I touched the upstream prometheus instance a few days ago 🙂 | 12:59:32 |
| das_j | Not by a lot actually. There's some stuff like nrUnsupportedSomething but the upstream exporter is mostly complete | 13:00:27 |
| das_j | Maybe you want --web.disable-exporter-metrics? :D | 13:00:49 |
|  Tseb joined the room. | 13:34:33 |
| @grahamc:nixos.org | nice | 14:15:18 |
| @grahamc:nixos.org | I'm hoping we could add a proper exporter directly to the evaluator and queue runner some day, but this looks great | 14:15:37 |
| das_j | yeah I was hoping the evaluator somehow dumps its state into the db as well but I couldn't find anything | 14:16:19 |
| @grahamc:nixos.org | one tricky bit of that is of course the process breakdown of hydra-evaulator calling hydra-evaluate-jobset calling hydra-eval-jobs, and potentially each of these could feasibly contain their own exporter, which would also be weird | 14:27:52 |
| das_j | What about this: The main daemon opens a fd for its subprocesses (FD 5 or something), and the subprocesses write their metrics there, maybe as kv pairs? Would be shitty but better than nothing | 14:29:04 |
| das_j | The main process can then collect these and serve them | 14:29:27 |
| @grahamc:nixos.org | I've tried to do that for ofborg in the past but was sort of a mess | 14:29:31 |
