| 7 Jul 2022 |
 7c6f434c | There are flaky tests, failed-before-anyway, disk full issues, timeouts… | 04:18:49 |
| 7c6f434c | (also complicated issues with some subset of the builders…) | 04:19:26 |
| 7c6f434c | Equating this to eval failures dilutes the clear «oops» status of reported definite-failures | 04:20:31 |
 Winter (she/her) | In reply to @7c6f434c:nitro.chat
Equating this to eval failures dilutes the clear «oops» status of reported definite-failures Wait, just to be clear: actual failures during a test run are reported as failures, just not eval errors? Or are they all reported as neutral? | 04:22:55 |
| 7c6f434c | evail failures are failures
| 04:23:20 |
| 7c6f434c | build/test failures are neutral | 04:23:28 |
| Winter (she/her) | ah, okay, thank you | 04:38:00 |
| Winter (she/her) | build/test failures i understand, but why can't eval failures be reported as failures? those shouldn't be problematic even in weird machines (i'd think...)
sorry for asking basically the same thing again | 04:39:19 |
| 7c6f434c | Eval failures are failures, yes | 04:39:49 |
| Winter (she/her) | test eval failures, though? | 05:49:30 |
| Winter (she/her) | like, when evaling a test(s) | 05:49:46 |
| 7c6f434c | Well, technically speaking pkg.tests not being there is a) eval failure b) definitely not a check failure… | 05:50:47 |
| Winter (she/her) | I mean when manually triggering a test and it's eval failing, that's reported as neutral | 05:51:21 |
| 7c6f434c | For example, a test missing would be an eval failure! | 05:51:44 |
| 7c6f434c | And possibly just an invocation typo | 05:51:58 |
| Winter (she/her) | Right, but that's reported as a neutral failure to GH. | 05:53:01 |
| 7c6f434c | As it should be | 05:53:13 |
| 7c6f434c | And also some tests might be assuming a platform and be useful as such and have eval failures on unexpected platforms | 05:53:58 |
| Winter (she/her) | I guess I'm confining my thinking to NixOS tests when we have different types of tests in-tree as well. | 05:54:38 |
| Winter (she/her) | (Not sure how widely used they actually are, though.) | 05:54:50 |
| 7c6f434c | Drawing a line is hard, and failure reports should be clear failures | 05:54:52 |
| 7c6f434c | NixOS tests might have some … interesting nuances between x86_64 and aarch64 | 05:55:28 |
| Winter (she/her) | How so, in what way? Obviously it depends on the test, but what are you thinking of? | 05:56:19 |
| 7c6f434c | Using a package that has hand-optimised assembly in it and finally gets marked broken on aarch64 when someone gets fed up with build failures | 05:57:14 |
| 7c6f434c | ofBorg is and should be conservative with the red marks | 05:58:16 |
| Winter (she/her) | Ah, yeah, in the case of broken packages, that won't stop manual triggers from trying to eval aarch64 (although we could gate passthru.tests based on broken for the automatic triggers) | 05:58:54 |
| Winter (she/her) | Yeah that makes sense | 05:58:59 |
| Winter (she/her) | Apologies for going in circles | 05:59:04 |
| 7c6f434c | Realistically GitHub is just too restricted to have good high-resolution status reporting | 06:00:11 |
| 7c6f434c | So ofBorg reports as red just the things that can be defined without finely carved logic | 06:00:37 |
