| 27 Oct 2022 |
 @davidak:matrix.org | i professionally work in QA and our team does manual tests for release und automate testing where possible. i think that belongs in one team, to have a common goal. there can still be people that only work in one area and maybe separate chat rooms if it helps decluttering, but we should coordinate the efforts more | 14:42:14 |
|  chreekat joined the room. | 14:46:13 |
|  Wanja Hentze joined the room. | 14:48:09 |
| @davidak:matrix.org | i just do my own thing and have my own standards for quality (often too perfectionistic). i would like to have a consensus on what level of quality we work towards and a combined effort on that. maybe even policies or recommendations. it's often unclear to me what and to i do things. so i try to learn how it is done or propose something and hope the reviewers are satisfied. i got a lot of such into the manual, like "how to deprecate packages" | 14:49:16 |
| Wanja Hentze | I think VM-based NixOS tests are seriously cool | 14:50:08 |
| Wanja Hentze | We have a few at work but almost nobody knows how to properly write them | 14:50:26 |
| Wanja Hentze | the ones who do learned by looking at random nixpgks code | 14:51:00 |
| @davidak:matrix.org | * i just do my own thing and have my own standards for quality (often too perfectionistic). i would like to have a consensus on what level of quality we work towards and a combined effort on that. maybe even policies or recommendations. it's often unclear to me what and how to do things. so i try to learn how it is done or propose something and hope the reviewers are satisfied. i got a lot of such into the manual, like "how to deprecate packages" | 14:52:23 |
| Wanja Hentze | also, maybe a NixOS Containers based variant could make sense. for when you're testing rather high-level things and test performance is more important than precise control over kernel version | 14:54:58 |
|  nbp joined the room. | 14:57:41 |
| Wanja Hentze | has anybody tried integrating GUI test automation like appium or robot framework into these? would it make sense? | 14:58:29 |
| @davidak:matrix.org | maybe i should bring up the idea of a QA team in the forum? | 14:59:28 |
| nbp | Robert Hensing (roberth): I had one question about testing, which is: Are nixpkgs/lib tests ever executed? When I was working on modules I added them to the release.nix files of Nixpkgs, but never saw it being reported on Hydra. | 15:01:20 |
| Wanja Hentze | In reply to @davidak:matrix.org
maybe i should bring up the idea of a QA team in the forum? would that be QA-for-nixpkgs or using-nix-for-QA? (or both) | 15:01:29 |
| @davidak:matrix.org | In reply to @whentze:matrix.org
has anybody tried integrating GUI test automation like appium or robot framework into these? would it make sense? i don't know of any container based tests. using testing frameworks make sense. we use language based test tools like pytest | 15:01:41 |
| @davidak:matrix.org | there is one gui test using that x11 framebuffer tool. j want to document that | 15:02:28 |
| Wanja Hentze | I have at some point made a nixos test that would start a single VM with a bunch of nixos containers in it. that also cut down the test runtime a bunch | 15:02:34 |
 Robert Hensing (roberth) | In reply to @nbp:mozilla.org
Robert Hensing (roberth): I had one question about testing, which is: Are nixpkgs/lib tests ever executed? When I was working on modules I added them to the release.nix files of Nixpkgs, but never saw it being reported on Hydra. there's a github action CI that runs nix-build --arg pkgs import ./. {} ./lib/tests/release.nix. I don't think there's a hydra job for it, but that doesn't seem necessary. | 15:03:51 |
| @davidak:matrix.org | In reply to @whentze:matrix.org
would that be QA-for-nixpkgs or using-nix-for-QA? (or both) a qa team for our project, nixos, nixpkgs etc | 15:04:01 |
| @davidak:matrix.org | * there is one gui test using that x11 framebuffer tool. i want to document that | 15:04:50 |
| Wanja Hentze | does nixpkgs (the repo) implement the "no rocket science" rule btw? | 15:04:51 |
| Wanja Hentze | the one that says CI on main must always be green, no exceptions, and this should be automatically enforced | 15:05:25 |
|  DavHau joined the room. | 15:06:04 |
| Robert Hensing (roberth) | In reply to @whentze:matrix.org
does nixpkgs (the repo) implement the "no rocket science" rule btw? generally, no. It is too large to test everything all the time. We do have some guided checks with passthru.tests or ofborg commands though | 15:06:46 |
| Wanja Hentze | is there a way to automatically deduce which tests could even theoretically be broken by a change? | 15:07:51 |
| Wanja Hentze | I feel like even `nix-build` almost gives that | 15:08:18 |
| Robert Hensing (roberth) | In reply to @whentze:matrix.org
would that be QA-for-nixpkgs or using-nix-for-QA? (or both) I like this distinction. This room is mostly for the latter, but we should definitely have one for QA-for-nixpkgs too | 15:08:24 |
| @davidak:matrix.org | In reply to @whentze:matrix.org
is there a way to automatically deduce which tests could even theoretically be broken by a change? nixpkgs-review builds reverse dependencies, but not their tests i think | 15:08:54 |
| Wanja Hentze | would it make sense to employ a "rollup" type workflow to reduce the cost of having all tests green always? | 15:09:42 |
