| 7 Feb 2024 |
 Dominic Mills | In reply to @fricklerhandwerk:matrix.org
Dominic Mills: Hey, it would be great if you kicked off the process with a CFP, because we should definitely take the opportunity to get funding.
How far do you want to go with it? Are you willing and available to drive it to conclusion: progress tracking, evaluation, final report, the whole project management long tail? (bearbeitet) * yes, I'm willing and available at do all of what you just mentioned for the duration of the programme. | 21:20:51 |
| 8 Feb 2024 |
|  symys joined the room. | 06:22:03 |
| symys changed their profile picture. | 18:46:54 |
| 9 Feb 2024 |
 @dooy:matrix.org | Is there any work towards creating an LLM bot that integrates Nix documentation for user support and generating specific documentation | 10:35:24 |
 fricklerhandwerk | In reply to @dooy:matrix.org
Is there any work towards creating an LLM bot that integrates Nix documentation for user support and generating specific documentation There were some internal experiments with commercial offers at Tweag in the past months, based on the source code, i.e. official documentation. Unsurprisingly the quality was not overwhelming, because many things are simply not written down.
Last year there was quite harsh opposition to mine Discourse and IRC/Matrix logs due to copyright/privacy concerns. We thought about implementing a self-hosted setup and offering it to the community for testing. But as those volunteer efforts go, not much has happened due to other priorities.
Ideally such an LLM would graze over Discourse, Matrix, GitHub, and the sources, and reply to questions with summaries with references. Would be great to have such a smart dumb search engine, because of those things that are written down somewhere, most are really hard to find manually.
| 10:49:44 |
| 11 Feb 2024 |
 @fractivore:cyberia.club | In reply to @dooy:matrix.org
Is there any work towards creating an LLM bot that integrates Nix documentation for user support and generating specific documentation In my experience LLMs are really awful at Nix right now and hallucinate a lot. I think significant advancement would probably need to happen for this to be helpful rather than confusing, as a bot pulling from existing LLMs is very likely to hallucinate and provide incorrect documentation. Probably, an LLM would need to be trained specifically for the task, and yeah I feel like the best one could even hope for right now is for the LLM to provide links to relevant discussions as described above, or be able to tell you "this is new territory" with some level of confidence. | 18:42:12 |
| @fractivore:cyberia.club | This also brings up the inevitable question of training ethics. Is there a means for indicating consent to train LLMs on, for example, code in nixpkgs? Is that up to the maintainer, or what? How does that work for discourse? | 18:46:07 |
| @fractivore:cyberia.club | My bias on this is, I'm a defender of "ai" tech, but also, people are thinking it's way better and more accurate right now than it actually is due to a small number of impressive cases like Alpha Go, and also the ethics are in a spot where nobody knows quite what to do and it's very easy to piss people off | 18:47:50 |
| @fractivore:cyberia.club | All that being said, docs are one place that LLMs should have some good use-cases, so I think it's good to keep exploring those. | 18:49:18 |
| @fractivore:cyberia.club | Finding relevant discussions would be more of a classifier than a generator | 18:51:19 |
| @dooy:matrix.org | If it could encapsulate the function logic without merely skimming over the consent code, that would be ideal. However, my knowledge in this area is limited. ChatGPT has been immensely helpful to me over the past year as I've been getting up to speed with NixOS. For the first time in my life, after being a hobbyist for 15 years, I've started working on my first PR. It assists me in understanding the function logic and already possesses a substantial amount of knowledge about Nix too. The value it provides is immense, and I feel significantly less stressed than before. It feels like I'm moving quickly, which empowers me.
| 21:24:30 |
| 12 Feb 2024 |
| symys | Sounds like it's gotten a little better since the last time I tried it. | 01:41:47 |
| symys | But... You haven't run in to situations where it lies to you? 🤨 Cuz that's the thing that would make it dubious to use unsupervised in docs. | 01:43:34 |
| symys | Personally I encounter hallucinations nearly every time I use chatGPT. | 01:45:04 |
| symys | Could be clearly labeled with a warning tho, I spose | 02:02:09 |
| symys | Personally I feel like a q&a support bot with warning labels would make more sense than a doc generator as such. | 02:05:21 |
| symys | I've been on the other hand thinking about "deterministic" doc generation... Which has certain advantages over "AI" (namely accuracy) but also distinct issues (verbosity). | 02:11:58 |
