| 27 Oct 2022 |
 @davidak:matrix.org | * there is one gui test using that x11 framebuffer tool. i want to document that | 15:04:50 |
 Wanja Hentze | does nixpkgs (the repo) implement the "no rocket science" rule btw? | 15:04:51 |
| Wanja Hentze | the one that says CI on main must always be green, no exceptions, and this should be automatically enforced | 15:05:25 |
|  DavHau joined the room. | 15:06:04 |
 Robert Hensing (roberth) | In reply to @whentze:matrix.org
does nixpkgs (the repo) implement the "no rocket science" rule btw? generally, no. It is too large to test everything all the time. We do have some guided checks with passthru.tests or ofborg commands though | 15:06:46 |
| Wanja Hentze | is there a way to automatically deduce which tests could even theoretically be broken by a change? | 15:07:51 |
| Wanja Hentze | I feel like even `nix-build` almost gives that | 15:08:18 |
| Robert Hensing (roberth) | In reply to @whentze:matrix.org
would that be QA-for-nixpkgs or using-nix-for-QA? (or both) I like this distinction. This room is mostly for the latter, but we should definitely have one for QA-for-nixpkgs too | 15:08:24 |
| @davidak:matrix.org | In reply to @whentze:matrix.org
is there a way to automatically deduce which tests could even theoretically be broken by a change? nixpkgs-review builds reverse dependencies, but not their tests i think | 15:08:54 |
| Wanja Hentze | would it make sense to employ a "rollup" type workflow to reduce the cost of having all tests green always? | 15:09:42 |
| Wanja Hentze | (I have only a little experience with the nixpkgs repo but a bunch with rust-lang/rust so that's what I'm used to) | 15:10:27 |
| Robert Hensing (roberth) | In reply to @whentze:matrix.org
would it make sense to employ a "rollup" type workflow to reduce the cost of having all tests green always? The staging workflow is kind of like this. A potential problem is that we basically never have an all-green situation. That's partly chick-and-egg, partly because Nixpkgs is huge. | 15:13:01 |
| Robert Hensing (roberth) | The automation would have to deal with false positives somehow and/or we'd have to fix all flaky tests | 15:13:53 |
| @davidak:matrix.org | we could mark tests as flaky, so they are ignored until the status is resolved. this is how the admin team i worked in handled false positives in our monitoring... few tests where ignored for years, because we don't had time to fix them | 15:18:49 |
| @davidak:matrix.org | the tests did still check every minute, but did not appear on the status monitor and did not send notigications. there was a specific list for them | 15:20:11 |
| @davidak:matrix.org | * the tests did still check every minute, but did not appear on the status monitor and did not send notifications. there was a specific list for them | 15:21:15 |
 nbp | If this is testing using Nix, one thing that is missing (or I am not aware that this was fixed) is that today, Nix needs a special set of environment variables to setup a testing environment.
(see https://github.com/NixOS/nixpkgs/blob/master/lib/tests/release.nix#L20-L34 )
This could be abstracted either with a command-line configuration file for all nix commands, or with a way a new nix tests --init, nix tests run <command running nix tests> or which output the set of environment variables to be used. | 15:22:02 |
| Robert Hensing (roberth) | I feel like the always green CI issue has been discussed many times before, but I couldn't find an issue on nixpkgs. A more persistent medium like that may be more productive for this; gather knowledge, coordinating experiments and development. | 15:26:22 |
|  [0x4A6F] joined the room. | 15:27:04 |
| Robert Hensing (roberth) | nbp: seems like a valid issue to raise in the nix project | 15:28:25 |
| Robert Hensing (roberth) | I don't know if nix tests --init is the right solution, partly because environment variables require a little dance to load into the calling shell | 15:29:40 |
| Robert Hensing (roberth) | running Nix inside the sandbox needs better support anyway because of RFC 92, computed derivations | 15:30:54 |
|  danielbarter joined the room. | 15:34:22 |
|  Profpatsch joined the room. | 15:42:53 |
|  alejandrosame joined the room. | 16:39:43 |
|  infinisil joined the room. | 16:47:21 |
|  Jan Tojnar joined the room. | 17:50:19 |
|  pinage404 joined the room. | 21:50:04 |
| 28 Oct 2022 |
|  olaf joined the room. | 16:21:54 |
| 9 Nov 2022 |
| Robert Hensing (roberth) | Would someone like to review testBuildFailure and testEqualContents? https://github.com/NixOS/nixpkgs/pull/196251 | 12:56:02 |
