| 26 Sep 2024 |
 mattleon | That's a great question, I got my information from this medium post on the OG llama 3 with 8B parameters. The gif of the terminal output of running an inference indicates the number of tokens per second at the end:
https://medium.com/@benoit.clouet/running-llama3-on-the-gpu-of-a-rk1-turing-pi-6dddb9e14521
I do wonder if hass exposes a performance graph 🤔
In either case, a 3060 is quite a bit more powerful than what I'm working with | 14:54:15 |
 K900 | The RK3588 has an NPU | 14:54:37 |
| K900 | Which is presumably currently not used | 14:54:42 |
| K900 | Because nothing supports it | 14:54:45 |
| K900 | But work is ongoing on that | 14:54:53 |
| mattleon | This reddit thread has some performance notes for various models and Nvidia cards, but I don't see llama with 3b parameters: https://www.reddit.com/r/LocalLLaMA/comments/13j5cxf/how_many_tokens_per_second_do_you_guys_get_with/ | 14:56:09 |
 @hexa:lossy.network | In reply to @mattleon:matrix.org
This reddit thread has some performance notes for various models and Nvidia cards, but I don't see llama with 3b parameters: https://www.reddit.com/r/LocalLLaMA/comments/13j5cxf/how_many_tokens_per_second_do_you_guys_get_with/ I think the smallest model was 4 or 8b thus far | 15:07:51 |
 Mackeveli | Is there some way of overriding cacert and adding private CAs to it in the music-assistant package like in the home-assistant package?
https://wiki.nixos.org/wiki/Home_Assistant#Trust_a_private_certificate_authority | 16:18:19 |
| @hexa:lossy.network | replace home-assistant with music-assisant | 17:09:33 |
| @hexa:lossy.network | both of them consume certifi, the package handing out cacerts for python consumers | 17:09:48 |
| Mackeveli | I tried that but the music-assistant package doesn't expose a packageOverrides variable like the home-assistant package does so I get this error upon doing a nixos-rebuild:
error: function 'anonymous lambda' called with unexpected argument 'packageOverrides'
at /nix/store/y6205wq8hxvpqvl8l9d1n9xah01kg0lq-source/pkgs/by-name/mu/music-assistant/package.nix:1:1:
1| { lib
| ^
2| , python3
| 17:27:06 |
| @hexa:lossy.network | ah, hm. | 17:27:46 |
| @hexa:lossy.network | you can pass an overriden cacert by passing the NIX_SSL_CERT_FILE environment variable to the systemd unit then | 17:28:54 |
| @hexa:lossy.network | systemd.services.music-assistant.environment.NIX_SSL_CERT_FILE = "${pkgs.cacert.override { extraCertificateFiles [ ./foo.pem ] }}/etc/ssl/certs/ca-bundle.crt";
| 17:30:57 |
| @hexa:lossy.network | * systemd.services.music-assistant.environment = {
NIX_SSL_CERT_FILE = "${pkgs.cacert.override { extraCertificateFiles [ ./foo.pem ] }}/etc/ssl/certs/ca-bundle.crt";
};
| 17:31:08 |
| @hexa:lossy.network | also supports extraCertificateStrings, a list of strings | 17:34:16 |
| Mackeveli | Hmm I tried that but now I'm getting an unrelated error upon running nixos-rebuild:
error: attempt to call something which is not a function but a list
at /nix/store/y6205wq8hxvpqvl8l9d1n9xah01kg0lq-source/pkgs/by-name/mu/music-assis
tant/package.nix:20:52:
19| providerNames = lib.attrNames providerPackages;
20| providerDependencies = lib.concatMap (provider: (providerPackages.${provider} python.pkgs)) providers;
| ^
21|
| 17:34:41 |
| @hexa:lossy.network | post your config | 17:36:48 |
| Mackeveli | https://github.com/negatethis/nixos-config/blob/main/services/music-assistant.nix | 18:14:15 |
| @hexa:lossy.network | ok, I see the issue | 18:18:42 |
| @hexa:lossy.network | also note that we are lacking dependencies for some of these providers | 18:20:18 |
| @hexa:lossy.network | https://github.com/NixOS/nixpkgs/pull/344704 | 18:23:35 |
| Mackeveli | Gotcha. I'll stick to the music-assistant container and use only HTTP while this is sorted out. I appreciate the help! | 18:34:58 |
| 27 Sep 2024 |
 Sandro 🐧 | hexa: did you already think or investigate if we can follow ha with replacing pip with uv? https://developers.home-assistant.io/blog/2024/04/03/build-images-with-uv/ | 12:33:30 |
| @hexa:lossy.network | we don't use pip? | 12:33:58 |
| @hexa:lossy.network | i'm not sure what you're asking for | 12:34:46 |
| @hexa:lossy.network | https://github.com/home-assistant/core/commit/c8948e3b1c8b1f52df6185695331cf6471c86829 | 12:39:44 |
| @hexa:lossy.network | probably what he is talking about | 12:40:36 |
| Sandro 🐧 | oh yeah, right 🤦♂️ | 15:09:33 |
| Sandro 🐧 | Would it make sense to replace it here https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/interpreters/python/hooks/pip-install-hook.sh#L14 ?
Is that even a part which takes a while to run? | 15:10:54 |
