 | Nix Documentation | 420 Members |
| Discussion about documentation improvements around the Nix ecosystem | 87 Servers |
| Nix Documentation | 420 Members |
| Discussion about documentation improvements around the Nix ecosystem | 87 Servers |
| Sender | Message | Time |
|---|---|---|
| 9 Feb 2024 | ||
 fricklerhandwerk | In reply to @dooy:matrix.org There were some internal experiments with commercial offers at Tweag in the past months, based on the source code, i.e. official documentation. Unsurprisingly the quality was not overwhelming, because many things are simply not written down. Last year there was quite harsh opposition to mine Discourse and IRC/Matrix logs due to copyright/privacy concerns. We thought about implementing a self-hosted setup and offering it to the community for testing. But as those volunteer efforts go, not much has happened due to other priorities. Ideally such an LLM would graze over Discourse, Matrix, GitHub, and the sources, and reply to questions with summaries with references. Would be great to have such a smart dumb search engine, because of those things that are written down somewhere, most are really hard to find manually. | 10:49:44 |
| 11 Feb 2024 | ||
 @fractivore:cyberia.club | In reply to @dooy:matrix.orgIn my experience LLMs are really awful at Nix right now and hallucinate a lot. I think significant advancement would probably need to happen for this to be helpful rather than confusing, as a bot pulling from existing LLMs is very likely to hallucinate and provide incorrect documentation. Probably, an LLM would need to be trained specifically for the task, and yeah I feel like the best one could even hope for right now is for the LLM to provide links to relevant discussions as described above, or be able to tell you "this is new territory" with some level of confidence. | 18:42:12 |
| @fractivore:cyberia.club | This also brings up the inevitable question of training ethics. Is there a means for indicating consent to train LLMs on, for example, code in nixpkgs? Is that up to the maintainer, or what? How does that work for discourse? | 18:46:07 |
| @fractivore:cyberia.club | My bias on this is, I'm a defender of "ai" tech, but also, people are thinking it's way better and more accurate right now than it actually is due to a small number of impressive cases like Alpha Go, and also the ethics are in a spot where nobody knows quite what to do and it's very easy to piss people off | 18:47:50 |
| @fractivore:cyberia.club | All that being said, docs are one place that LLMs should have some good use-cases, so I think it's good to keep exploring those. | 18:49:18 |
| @fractivore:cyberia.club | Finding relevant discussions would be more of a classifier than a generator | 18:51:19 |
 @dooy:matrix.org | If it could encapsulate the function logic without merely skimming over the consent code, that would be ideal. However, my knowledge in this area is limited. ChatGPT has been immensely helpful to me over the past year as I've been getting up to speed with NixOS. For the first time in my life, after being a hobbyist for 15 years, I've started working on my first PR. It assists me in understanding the function logic and already possesses a substantial amount of knowledge about Nix too. The value it provides is immense, and I feel significantly less stressed than before. It feels like I'm moving quickly, which empowers me. | 21:24:30 |
| 12 Feb 2024 | ||
 symys | Sounds like it's gotten a little better since the last time I tried it. | 01:41:47 |
| symys | But... You haven't run in to situations where it lies to you? 🤨 Cuz that's the thing that would make it dubious to use unsupervised in docs. | 01:43:34 |
| symys | Personally I encounter hallucinations nearly every time I use chatGPT. | 01:45:04 |
| symys | Could be clearly labeled with a warning tho, I spose | 02:02:09 |
| symys | Personally I feel like a q&a support bot with warning labels would make more sense than a doc generator as such. | 02:05:21 |
| symys | I've been on the other hand thinking about "deterministic" doc generation... Which has certain advantages over "AI" (namely accuracy) but also distinct issues (verbosity). | 02:11:58 |
| symys | * I've been on the other hand thinking about "deterministic" doc generation... Which has certain advantages over "AI" (namely accuracy) but also distinct issues (verbosity and style). | 02:12:27 |
| symys | * I've been on the other hand thinking about "deterministic" doc generation... Which has certain advantages over "AI" (namely accuracy) but also distinct issues (overly verbose, formulaic style, always low-level). | 02:16:31 |
| symys | * I've been on the other hand thinking about "deterministic" doc generation... Which has certain advantages over "AI" (namely accuracy) but also distinct issues (overly verbose, formulaic style, unable to recontexutalize or "zoom out"). | 02:17:08 |
| 13 Feb 2024 | ||
 @adam:robins.wtf | hallucinations aren't great, but humans can provide incorrect information too, and it seems the alternative is going to chat/forum and asking humans to help. granted, that would be in a public forum where other people could correct the bad information | 18:17:19 |
 @jade_:matrix.org | https://tenforward.social/@noracodes/111926120933093851 | 21:15:01 |
| @jade_:matrix.org | we should have a glossary of common top level Nix expression shapes | 21:15:21 |
| @jade_:matrix.org | and how to recognize them | 21:15:26 |
| fricklerhandwerk | In reply to @jade_:matrix.org Opened an issue so we don’t forget https://github.com/NixOS/nix.dev/issues/918 | 21:29:34 |
| 14 Feb 2024 | ||
| @fractivore:cyberia.club | In reply to @adam:robins.wtfI guess generating docs could work if they get a PR and are reviewed just like human additions. It could be set to do it kinda slowly, like a doc a day, to reduce alert fatigue on that. | 15:52:29 |
| 15 Feb 2024 | ||
 pkharvey | 08:13:00 | |
 Jan Tojnar | Could anyone please check https://github.com/NixOS/nix-pills/pull/228#issuecomment-1945590743 | 09:05:22 |
| Jan Tojnar | I guess I have reviewed it too authoritatively and they closed it. | 09:06:03 |
 infinisil | I'll be in the docs team meeting in a couple minutes | 15:00:53 |
| infinisil | (still a couple more) | 15:06:05 |
 i97henka | Sorry couldn't make it today - will join again soon I hope. | 15:28:45 |
| infinisil | No worries :) | 17:22:22 |
| 19 Feb 2024 | ||
 Soenke Klinger joined the room. | 18:18:11 | |