| 1 Oct 2023 |
 hexa | https://ofborg.org/prometheus/alerts hmm | 14:41:55 |
| hexa | don't think we monitor the darwin machines in prometheus | 14:43:41 |
 Lily Foster | In reply to @hexa:lossy.network
don't think we monitor the darwin machines in prometheus We do, but I bet they don't push that metric | 14:45:15 |
| hexa | https://ofborg.org/prometheus/graph?g0.expr=node_os_info&g0.tab=1&g0.stacked=0&g0.show_exemplars=0&g0.range_input=1h | 14:45:36 |
| hexa | I don't think we do 😄 | 14:46:07 |
| Lily Foster | Yeah looks like they are only sending ofborg metrics | 14:46:19 |
| Lily Foster | No system metrics | 14:46:23 |
| hexa | oh | 14:46:30 |
| hexa | where did you find a darwin metric? | 14:46:51 |
| Lily Foster | Actually the metrics are only sent from the central queue manager. So yeah they aren't sending anything | 14:47:18 |
| Lily Foster | (for darwin build info) | 14:47:25 |
| Lily Foster | You're right | 14:47:29 |
|  pbsds joined the room. | 14:49:17 |
 cole-h | In reply to @cafkafk:gitter.im
disk full https://github.com/NixOS/nixpkgs/pull/258395/checks?check_run_id=17292775692 Should be fixed now. | 14:56:40 |
| Lily Foster | Any chance we could run the prometheus agent on darwin cole-h? 🥺 | 14:57:27 |
| cole-h | We do, I guess they're just not scraped | 14:58:43 |
| cole-h | curl -ss http://...:9100/metrics | head
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.9458e-05
go_gc_duration_seconds{quantile="0.25"} 5.4583e-05
go_gc_duration_seconds{quantile="0.5"} 6.6707e-05
go_gc_duration_seconds{quantile="0.75"} 9.1292e-05
go_gc_duration_seconds{quantile="1"} 0.000423
go_gc_duration_seconds_sum 251.971410887
go_gc_duration_seconds_count 115795
# HELP go_goroutines Number of goroutines that currently exist.
| 15:00:39 |
| cole-h | OK, they're scraped now. | 15:14:17 |
| hexa | the node_filesystem_free metric that is used in the alerting rule does not exist | 15:31:28 |
| hexa | 
Download image.png | 15:31:52 |
| hexa | this one | 15:31:53 |
| cole-h | 🔥🐶🔥 | 15:32:09 |
| hexa | should probably use avail_bytes 😄 | 15:32:17 |
| hexa | or free_bytes | 15:32:34 |
| hexa | heck, what is the difference? | 15:32:40 |
| cole-h | compression or something maybe? those 2 queries differ on the linux boxes but not the darwin ones | 15:33:21 |
| cole-h | wait nvm it does differ on both but the difference is larger on linux | 15:33:48 |
| cole-h | https://github.com/prometheus/node_exporter/issues/269 🤷 | 15:34:28 |
| cole-h | hexa: I'm officially tasking you: tell me which one I should use, free or avail? 😆 | 15:34:50 |
| hexa | cool. | 15:35:00 |
