| 11 Feb 2025 |
 motiejus | Filed https://github.com/NixOS/nixpkgs/issues/381280 for visibility | 21:13:08 |
| motiejus | * Filed https://github.com/NixOS/nixpkgs/issues/381280 | 21:13:18 |
 @hexa:lossy.network | yeah, well | 21:19:52 |
| @hexa:lossy.network | I'm here | 21:19:53 |
| @hexa:lossy.network | I'm not quite sure where to start debugging this further | 21:20:08 |
| @hexa:lossy.network | multiprocessing is a python standard library module | 21:20:55 |
| @hexa:lossy.network | our python is not compiled with pgo | 21:21:04 |
 K900 | I would start by attaching a profiler | 21:21:28 |
| @hexa:lossy.network | but the problem could also be gomp | 21:21:33 |
| K900 | Like py-spy | 21:21:43 |
| K900 | And then maybe perf if it's not in Python code | 21:21:50 |
| @hexa:lossy.network | there was a perf already | 21:21:59 |
| @hexa:lossy.network | https://github.com/blakeblackshear/frigate/discussions/16450#discussioncomment-12143059 | 21:22:13 |
| K900 | I'm pretty sure all the interesting bits are too small to see on this graph | 21:23:35 |
| @hexa:lossy.network | yeah, agreed | 21:23:42 |
| K900 | Because gompbarrierwait_end just waits on a bunch of threads | 21:24:19 |
| K900 | https://github.com/gcc-mirror/gcc/blob/master/libgomp/config/posix/bar.c#L73-L105 | 21:24:20 |
| @hexa:lossy.network | motiejus: so can you zoom into what happens before that ends? | 21:25:39 |
| K900 | Also, which threads was that even recorded form | 21:25:56 |
| K900 | * Also, which threads was that even recorded from | 21:25:58 |
| K900 | I wonder if we're just not seeing any of the worker threads | 21:26:04 |
| @hexa:lossy.network | I'm about to set up frigate on an n100 machine with a coral m.2 card | 21:27:05 |
| @hexa:lossy.network | so I can probably try and reproduce this in a few days | 21:27:13 |
| 12 Feb 2025 |
| motiejus | Uploaded full svg: https://github.com/NixOS/nixpkgs/issues/381280#issuecomment-2652896354
Good point re worker threads, I'll see if there are any, and profile those. | 07:45:51 |
| motiejus | Unrelated: can I start lobbying a CFLAG -fno-omit-frame-pointer, at least on x86_64-linux? So I don't need to do such partial hacks. Some background reading by Brendan Gregg. | 07:48:14 |
| motiejus | Relevant quote:
Fedora and Ubuntu have now returned frame pointers, which is great news. People should start running these releases in 2024 and will find that CPU flame graphs make more sense, Off-CPU flame graphs work for the first time, and other new things become possible. It's also a win for continuous profilers, as they don't need to convince their customers to make OS changes to get profiles to fully work.
As you can see, this debugging session would benefit from those greatly. :)
| 07:49:27 |
| motiejus | * Relevant quote:
Fedora and Ubuntu have now returned frame pointers, which is great news. People should start running these releases in 2024 and will find that CPU flame graphs make more sense, Off-CPU flame graphs work for the first time, and other new things become possible. It's also a win for continuous profilers, as they don't need to convince their customers to make OS changes to get profiles to fully work.
As you can see, this debugging session would benefit from those greatly.
| 08:02:37 |
| motiejus | Uploaded a bunch of flame graphs, probably including worker nodes (if I got the perf command right). | 08:26:07 |
| motiejus | * Uploaded a bunch of flame graphs, probably including worker threads (if I got the perf command right). | 08:26:25 |
| K900 | In reply to@motiejus:jakstys.lt
Unrelated: can I start lobbying a CFLAG -fno-omit-frame-pointer, at least on x86_64-linux? So I don't need to do such partial hacks. Some background reading by Brendan Gregg. We tend to use debuginfo for this instead | 08:57:01 |
