| 27 Oct 2022 |
 Wanja Hentze | the ones who do learned by looking at random nixpgks code | 14:51:00 |
 @davidak:matrix.org | * i just do my own thing and have my own standards for quality (often too perfectionistic). i would like to have a consensus on what level of quality we work towards and a combined effort on that. maybe even policies or recommendations. it's often unclear to me what and how to do things. so i try to learn how it is done or propose something and hope the reviewers are satisfied. i got a lot of such into the manual, like "how to deprecate packages" | 14:52:23 |
| Wanja Hentze | also, maybe a NixOS Containers based variant could make sense. for when you're testing rather high-level things and test performance is more important than precise control over kernel version | 14:54:58 |
|  nbp joined the room. | 14:57:41 |
| Wanja Hentze | has anybody tried integrating GUI test automation like appium or robot framework into these? would it make sense? | 14:58:29 |
| @davidak:matrix.org | maybe i should bring up the idea of a QA team in the forum? | 14:59:28 |
| nbp | Robert Hensing (roberth): I had one question about testing, which is: Are nixpkgs/lib tests ever executed? When I was working on modules I added them to the release.nix files of Nixpkgs, but never saw it being reported on Hydra. | 15:01:20 |
| Wanja Hentze | In reply to @davidak:matrix.org
maybe i should bring up the idea of a QA team in the forum? would that be QA-for-nixpkgs or using-nix-for-QA? (or both) | 15:01:29 |
| @davidak:matrix.org | In reply to @whentze:matrix.org
has anybody tried integrating GUI test automation like appium or robot framework into these? would it make sense? i don't know of any container based tests. using testing frameworks make sense. we use language based test tools like pytest | 15:01:41 |
| @davidak:matrix.org | there is one gui test using that x11 framebuffer tool. j want to document that | 15:02:28 |
| Wanja Hentze | I have at some point made a nixos test that would start a single VM with a bunch of nixos containers in it. that also cut down the test runtime a bunch | 15:02:34 |
 Robert Hensing (roberth) | In reply to @nbp:mozilla.org
Robert Hensing (roberth): I had one question about testing, which is: Are nixpkgs/lib tests ever executed? When I was working on modules I added them to the release.nix files of Nixpkgs, but never saw it being reported on Hydra. there's a github action CI that runs nix-build --arg pkgs import ./. {} ./lib/tests/release.nix. I don't think there's a hydra job for it, but that doesn't seem necessary. | 15:03:51 |
| @davidak:matrix.org | In reply to @whentze:matrix.org
would that be QA-for-nixpkgs or using-nix-for-QA? (or both) a qa team for our project, nixos, nixpkgs etc | 15:04:01 |
| @davidak:matrix.org | * there is one gui test using that x11 framebuffer tool. i want to document that | 15:04:50 |
| Wanja Hentze | does nixpkgs (the repo) implement the "no rocket science" rule btw? | 15:04:51 |
| Wanja Hentze | the one that says CI on main must always be green, no exceptions, and this should be automatically enforced | 15:05:25 |
|  DavHau joined the room. | 15:06:04 |
| Robert Hensing (roberth) | In reply to @whentze:matrix.org
does nixpkgs (the repo) implement the "no rocket science" rule btw? generally, no. It is too large to test everything all the time. We do have some guided checks with passthru.tests or ofborg commands though | 15:06:46 |
| Wanja Hentze | is there a way to automatically deduce which tests could even theoretically be broken by a change? | 15:07:51 |
| Wanja Hentze | I feel like even `nix-build` almost gives that | 15:08:18 |
| Robert Hensing (roberth) | In reply to @whentze:matrix.org
would that be QA-for-nixpkgs or using-nix-for-QA? (or both) I like this distinction. This room is mostly for the latter, but we should definitely have one for QA-for-nixpkgs too | 15:08:24 |
| @davidak:matrix.org | In reply to @whentze:matrix.org
is there a way to automatically deduce which tests could even theoretically be broken by a change? nixpkgs-review builds reverse dependencies, but not their tests i think | 15:08:54 |
| Wanja Hentze | would it make sense to employ a "rollup" type workflow to reduce the cost of having all tests green always? | 15:09:42 |
| Wanja Hentze | (I have only a little experience with the nixpkgs repo but a bunch with rust-lang/rust so that's what I'm used to) | 15:10:27 |
| Robert Hensing (roberth) | In reply to @whentze:matrix.org
would it make sense to employ a "rollup" type workflow to reduce the cost of having all tests green always? The staging workflow is kind of like this. A potential problem is that we basically never have an all-green situation. That's partly chick-and-egg, partly because Nixpkgs is huge. | 15:13:01 |
| Robert Hensing (roberth) | The automation would have to deal with false positives somehow and/or we'd have to fix all flaky tests | 15:13:53 |
| @davidak:matrix.org | we could mark tests as flaky, so they are ignored until the status is resolved. this is how the admin team i worked in handled false positives in our monitoring... few tests where ignored for years, because we don't had time to fix them | 15:18:49 |
| @davidak:matrix.org | the tests did still check every minute, but did not appear on the status monitor and did not send notigications. there was a specific list for them | 15:20:11 |
| @davidak:matrix.org | * the tests did still check every minute, but did not appear on the status monitor and did not send notifications. there was a specific list for them | 15:21:15 |
| nbp | If this is testing using Nix, one thing that is missing (or I am not aware that this was fixed) is that today, Nix needs a special set of environment variables to setup a testing environment.
(see https://github.com/NixOS/nixpkgs/blob/master/lib/tests/release.nix#L20-L34 )
This could be abstracted either with a command-line configuration file for all nix commands, or with a way a new nix tests --init, nix tests run <command running nix tests> or which output the set of environment variables to be used. | 15:22:02 |
