| 14 Jul 2024 |
 nh2 | In reply to @imincik:matrix.org
I would be very happy if we could identify and disable couple of tests which are more likely to fail, but I would feel very uncomfortable to disable all of them. Ivan Mincik (imincik): Yes, we should not disable all tests just because some are flaky.
But we should disable all tests that are flaky in any way. Flaky tests cause huge problems: Disabling a large amount of nixpkgs packages, wasting time on Hydra and 1000x that time on users' computers, as well as contributor time.
Fixing flaky tests is an upstream responsibility; we should report the flakiness, disabled them in nixpkgs, and let upstream solve it (or help with that if we have the time). | 16:50:18 |
 Ivan Mincik (imincik) | In reply to @nh2:matrix.org
Ivan Mincik (imincik): Yes, we should not disable all tests just because some are flaky.
But we should disable all tests that are flaky in any way. Flaky tests cause huge problems: Disabling a large amount of nixpkgs packages, wasting time on Hydra and 1000x that time on users' computers, as well as contributor time.
Fixing flaky tests is an upstream responsibility; we should report the flakiness, disabled them in nixpkgs, and let upstream solve it (or help with that if we have the time). Fully agree | 17:18:15 |
| Ivan Mincik (imincik) | In reply to @nh2:matrix.org
Ivan Mincik (imincik): Yes, we should not disable all tests just because some are flaky.
But we should disable all tests that are flaky in any way. Flaky tests cause huge problems: Disabling a large amount of nixpkgs packages, wasting time on Hydra and 1000x that time on users' computers, as well as contributor time.
Fixing flaky tests is an upstream responsibility; we should report the flakiness, disabled them in nixpkgs, and let upstream solve it (or help with that if we have the time). Let's create PR where we start disabling all gdal tests which we see failing, keep it open for at least one month and then merge. Repeat if needed. What do you think? | 17:23:07 |
 l0b0 | In reply to @imincik:matrix.org
Let's create PR where we start disabling all gdal tests which we see failing, keep it open for at least one month and then merge. Repeat if needed. What do you think? How about creating a new PR every time someone notices a set of failing tests? That way we can disable flaky tests ASAP, rather than after an arbitrary time. | 20:34:24 |
| l0b0 | By the way, do we have some sort of simple upstream feedback mechanism? Passing flaky tests onto upstream should hopefully result in less flaky tests. | 20:36:07 |
| 15 Jul 2024 |
| Ivan Mincik (imincik) | In reply to @vengmark2:matrix.org
By the way, do we have some sort of simple upstream feedback mechanism? Passing flaky tests onto upstream should hopefully result in less flaky tests. Just reporting that. Shame on me, I just spent a week with gdal devs and completely forgot to discuss this issue. | 08:22:26 |
