In reply to @fricklerhandwerk:matrix.org
Yeah just ditch those references. Thanks a lot!

In reply to @olafklingt:matrix.org
I started to think/brainstorm about the NixOS manual again. And i remember that fricklerhandwerk recently said that he thinks "a lot of things in the manual should be on the wiki". What was did you had in mind with of that statement?

In reply to @olafklingt:matrix.org
this closed pr https://github.com/NixOS/nix.dev/pull/964 triggered my brainstorming mind to think about the possibility to have testable tutorials.
What if every tutorial is an annotated nixos-test?
For me the question is not focused on the style of annotation right now.
But rather if you have ideas why an attempt to realize this would be futile?
maybe to difficult to realize for potential content contributors (deteriorating)?
maybe to complex to implement correctly (endless poc...)?

I find this to be a bit of a rabbit hole, because in my opinion (which may as well be very wrong), all of the tooling we have for that is near-useless for our purposes. Which doesn't make it impossible to do anything about the problem of tutorials diverging from the code, but raises the question of how to approach it regardless.

There are three ways I currently see, and find all of them to be a stretch:

• Add NixOS tests that accompany tutorials. Problem: duplication and a mess of a setup
• Use https://github.com/OceanSprint/tesh to extract test cases from tutorials. This would cover many use cases, but the tool is not actively developed, and there are lots of open questions. I think it is worth trying out.
• Do something completely different, as sketched here: https://github.com/fricklerhandwerk/beast. Problem: This very unorthodox and requires building some infrastructure first, so there's high risk of diverting attention from what we may think is more important.

In reply to @olafklingt:matrix.org
this closed pr https://github.com/NixOS/nix.dev/pull/964 triggered my brainstorming mind to think about the possibility to have testable tutorials.
What if every tutorial is an annotated nixos-test?
For me the question is not focused on the style of annotation right now.
But rather if you have ideas why an attempt to realize this would be futile?
maybe to difficult to realize for potential content contributors (deteriorating)?
maybe to complex to implement correctly (endless poc...)?

I find this to be a bit of a rabbit hole, because in my opinion (which may as well be very wrong), all of the tooling we have for that is near-useless for our purposes. Which doesn't make it impossible to do anything about the problem of tutorials diverging from the code, but raises the question of how to approach it regardless.

There are three ways I currently see, and find all of them to be a stretch:

• Add NixOS tests that accompany tutorials. Problem: duplication and a mess of a setup, which needs to be maintained by hand.
• Use https://github.com/OceanSprint/tesh to extract test cases from tutorials. This would cover many use cases, but the tool is not actively developed, and there are lots of open questions. I think it is worth trying out.
• Do something completely different, as sketched here: https://github.com/fricklerhandwerk/beast. Problem: This very unorthodox and requires building some infrastructure first, so there's high risk of diverting attention from what we may think is more important.

In reply to @fricklerhandwerk:matrix.org

I find this to be a bit of a rabbit hole, because in my opinion (which may as well be very wrong), all of the tooling we have for that is near-useless for our purposes. Which doesn't make it impossible to do anything about the problem of tutorials diverging from the code, but raises the question of how to approach it regardless.

There are three ways I currently see, and find all of them to be a stretch:

• Add NixOS tests that accompany tutorials. Problem: duplication and a mess of a setup, which needs to be maintained by hand.
• Use https://github.com/OceanSprint/tesh to extract test cases from tutorials. This would cover many use cases, but the tool is not actively developed, and there are lots of open questions. I think it is worth trying out.
• Do something completely different, as sketched here: https://github.com/fricklerhandwerk/beast. Problem: This very unorthodox and requires building some infrastructure first, so there's high risk of diverting attention from what we may think is more important.

In reply to @djacu:matrix.org
fricklerhandwerk: infinisil I've created an issue for expanding and improving the module documentation on nix.dev. I would appreciate your and anyone else's feedback.
https://github.com/NixOS/nix.dev/issues/966

olaf: You might be interested in what I was proposing for the module tutorials update. It's probably not as comprehensive as what you have in mind but might be suitable for smaller tests that don't need the infrastructure of a full nixos-test.

For my workshop, I created lessons where the code was separate from the markdown and inserted at build time of the site. Each example was evaluated/built and the result was also injected into the markdown. Essentially creating a system where the site would not build if the examples failed to evaluate. I didn't do any checking to make sure the answer was correct but that could be added.

The way I did this was pretty hacky and sometimes cursed (and definitely not the way we should do it on nix.dev) but it's what I had to do given my time constraints. With more time I would have needed to run Nix inside of Nix or come up with some other solution.