| 25 Sep 2024 |
|  @beat_link:matrix.org left the room. | 12:39:30 |
 @hexa:lossy.network | llama3.2:3b-instruct-q8_0 works great for home-assistant | 21:55:27 |
| 26 Sep 2024 |
 @adam:robins.wtf | ooo, 3.2 | 00:00:23 |
| @adam:robins.wtf | looks like it's fairly well tuned for home assistant type workloads | 00:18:59 |
|  Fabián Heredia set a profile picture. | 01:16:00 |
 bbigras | Does it work with function calling? | 04:14:29 |
| @hexa:lossy.network | yes | 07:14:58 |
|  Mackeveli joined the room. | 14:02:43 |
 mattleon | May I ask what hardware you are running inferencing on and how many tokens per second it generates?
It seems the hardware I'm running hass on can do about 3 tokens/s, trying to figure out if that's enough | 14:17:50 |
| @hexa:lossy.network | In reply to @mattleon:matrix.org
May I ask what hardware you are running inferencing on and how many tokens per second it generates?
It seems the hardware I'm running hass on can do about 3 tokens/s, trying to figure out if that's enough 3060 12GB | 14:47:07 |
| @hexa:lossy.network | In reply to @hexa:lossy.network
3060 12GB How do I find out how many tokens? | 14:47:24 |
| mattleon | That's a great question, I got my information from this medium post on the OG llama 3 with 8B parameters. The gif of the terminal output of running an inference indicates the number of tokens per second at the end:
https://medium.com/@benoit.clouet/running-llama3-on-the-gpu-of-a-rk1-turing-pi-6dddb9e14521
I do wonder if hass exposes a performance graph 🤔
In either case, a 3060 is quite a bit more powerful than what I'm working with | 14:54:15 |
 K900 | The RK3588 has an NPU | 14:54:37 |
| K900 | Which is presumably currently not used | 14:54:42 |
| K900 | Because nothing supports it | 14:54:45 |
| K900 | But work is ongoing on that | 14:54:53 |
| mattleon | This reddit thread has some performance notes for various models and Nvidia cards, but I don't see llama with 3b parameters: https://www.reddit.com/r/LocalLLaMA/comments/13j5cxf/how_many_tokens_per_second_do_you_guys_get_with/ | 14:56:09 |
| @hexa:lossy.network | In reply to @mattleon:matrix.org
This reddit thread has some performance notes for various models and Nvidia cards, but I don't see llama with 3b parameters: https://www.reddit.com/r/LocalLLaMA/comments/13j5cxf/how_many_tokens_per_second_do_you_guys_get_with/ I think the smallest model was 4 or 8b thus far | 15:07:51 |
| Mackeveli | Is there some way of overriding cacert and adding private CAs to it in the music-assistant package like in the home-assistant package?
https://wiki.nixos.org/wiki/Home_Assistant#Trust_a_private_certificate_authority | 16:18:19 |
| @hexa:lossy.network | replace home-assistant with music-assisant | 17:09:33 |
| @hexa:lossy.network | both of them consume certifi, the package handing out cacerts for python consumers | 17:09:48 |
| Mackeveli | I tried that but the music-assistant package doesn't expose a packageOverrides variable like the home-assistant package does so I get this error upon doing a nixos-rebuild:
error: function 'anonymous lambda' called with unexpected argument 'packageOverrides'
at /nix/store/y6205wq8hxvpqvl8l9d1n9xah01kg0lq-source/pkgs/by-name/mu/music-assistant/package.nix:1:1:
1| { lib
| ^
2| , python3
| 17:27:06 |
| @hexa:lossy.network | ah, hm. | 17:27:46 |
| @hexa:lossy.network | you can pass an overriden cacert by passing the NIX_SSL_CERT_FILE environment variable to the systemd unit then | 17:28:54 |
| @hexa:lossy.network | systemd.services.music-assistant.environment.NIX_SSL_CERT_FILE = "${pkgs.cacert.override { extraCertificateFiles [ ./foo.pem ] }}/etc/ssl/certs/ca-bundle.crt";
| 17:30:57 |
| @hexa:lossy.network | * systemd.services.music-assistant.environment = {
NIX_SSL_CERT_FILE = "${pkgs.cacert.override { extraCertificateFiles [ ./foo.pem ] }}/etc/ssl/certs/ca-bundle.crt";
};
| 17:31:08 |
| @hexa:lossy.network | also supports extraCertificateStrings, a list of strings | 17:34:16 |
| Mackeveli | Hmm I tried that but now I'm getting an unrelated error upon running nixos-rebuild:
error: attempt to call something which is not a function but a list
at /nix/store/y6205wq8hxvpqvl8l9d1n9xah01kg0lq-source/pkgs/by-name/mu/music-assis
tant/package.nix:20:52:
19| providerNames = lib.attrNames providerPackages;
20| providerDependencies = lib.concatMap (provider: (providerPackages.${provider} python.pkgs)) providers;
| ^
21|
| 17:34:41 |
| @hexa:lossy.network | post your config | 17:36:48 |
| Mackeveli | https://github.com/negatethis/nixos-config/blob/main/services/music-assistant.nix | 18:14:15 |
