| 27 Oct 2022 |
 @davidak:matrix.org | In reply to @whentze:matrix.org
is there a way to automatically deduce which tests could even theoretically be broken by a change? nixpkgs-review builds reverse dependencies, but not their tests i think | 15:08:54 |
 Wanja Hentze | would it make sense to employ a "rollup" type workflow to reduce the cost of having all tests green always? | 15:09:42 |
| Wanja Hentze | (I have only a little experience with the nixpkgs repo but a bunch with rust-lang/rust so that's what I'm used to) | 15:10:27 |
 Robert Hensing (roberth) | In reply to @whentze:matrix.org
would it make sense to employ a "rollup" type workflow to reduce the cost of having all tests green always? The staging workflow is kind of like this. A potential problem is that we basically never have an all-green situation. That's partly chick-and-egg, partly because Nixpkgs is huge. | 15:13:01 |
| Robert Hensing (roberth) | The automation would have to deal with false positives somehow and/or we'd have to fix all flaky tests | 15:13:53 |
| @davidak:matrix.org | we could mark tests as flaky, so they are ignored until the status is resolved. this is how the admin team i worked in handled false positives in our monitoring... few tests where ignored for years, because we don't had time to fix them | 15:18:49 |
| @davidak:matrix.org | the tests did still check every minute, but did not appear on the status monitor and did not send notigications. there was a specific list for them | 15:20:11 |
| @davidak:matrix.org | * the tests did still check every minute, but did not appear on the status monitor and did not send notifications. there was a specific list for them | 15:21:15 |
 nbp — PTO | If this is testing using Nix, one thing that is missing (or I am not aware that this was fixed) is that today, Nix needs a special set of environment variables to setup a testing environment.
(see https://github.com/NixOS/nixpkgs/blob/master/lib/tests/release.nix#L20-L34 )
This could be abstracted either with a command-line configuration file for all nix commands, or with a way a new nix tests --init, nix tests run <command running nix tests> or which output the set of environment variables to be used. | 15:22:02 |
| Robert Hensing (roberth) | I feel like the always green CI issue has been discussed many times before, but I couldn't find an issue on nixpkgs. A more persistent medium like that may be more productive for this; gather knowledge, coordinating experiments and development. | 15:26:22 |
|  [0x4A6F] joined the room. | 15:27:04 |
