| 19 Sep 2025 |
 connor (burnt/out) (UTC-8) | Do any of you run into errors in git-minimal's test suite regularly? On all three of my machines I usually see a rotation of test failures like this:
Test Summary Report
-------------------
t0050-filesystem.sh (Wstat: 256 (exited 1) Tests: 13 Failed: 0)
TODO passed: 11
Non-zero exit status: 1
t2200-add-update.sh (Wstat: 256 (exited 1) Tests: 19 Failed: 1)
Failed test: 16
Non-zero exit status: 1
t7002-mv-sparse-checkout.sh (Wstat: 256 (exited 1) Tests: 22 Failed: 1)
Failed test: 22
Non-zero exit status: 1
t7800-difftool.sh (Wstat: 256 (exited 1) Tests: 95 Failed: 1)
Failed test: 45
Non-zero exit status: 1
Files=1003, Tests=27703, 47 wallclock secs ( 4.80 usr 1.95 sys + 243.89 cusr 662.43 csys = 913.07 CPU)
Result: FAIL
| 19:34:45 |
 Lun | Some of those overlap with https://github.com/NixOS/nixpkgs/pull/443250/files | 19:36:07 |
 Philip Taron (UTC-8) | The funny thing is that I never see these 👀
Both you and Tristan are on aarch64-linux, right? | 19:48:57 |
| connor (burnt/out) (UTC-8) | My three desktop builders are all x86_64 (one i9-13900k and two 7950X3D) | 19:49:44 |
| Philip Taron (UTC-8) | Hmm, that's mine as well (7950X). | 19:50:21 |
| Philip Taron (UTC-8) | I support moving the Git tests in their entirety to passthru.tests. We can run that suite when we bump Git. | 19:51:11 |
 VladimÃr ÄŒunát | And when those tests fail, do... what? | 19:53:03 |
| Philip Taron (UTC-8) | Investigate, the same as we do with systemd and its mostly-system tests on each bump. | 19:53:57 |
| Lun | I wonder if it's related to what FS the build dir is on | 19:54:57 |
| Philip Taron (UTC-8) | Having a laundry list of tests that someone determined as being "flaky" without upstream correspondance doesn't seem that great. I suspect it has something to do with my use of ext4 why they're not flaky. | 19:55:30 |
| Philip Taron (UTC-8) | * Having a laundry list of tests that someone determined as being "flaky" without upstream correspondance doesn't seem that great. I suspect it has something to do with my use of ext4 why they're not flaky for me. | 19:55:38 |
| Philip Taron (UTC-8) | Jinx Lun :-) | 19:55:51 |
 hexa | among other things, yes, but also concurrency | 19:55:52 |
| connor (burnt/out) (UTC-8) | I do use ZFS on all three of them. I just tried to build on all three machines:
i9-13900K:
Test Summary Report
-------------------
t0050-filesystem.sh (Wstat: 256 (exited 1) Tests: 13 Failed: 0)
TODO passed: 11
Non-zero exit status: 1
t7513-interpret-trailers.sh (Wstat: 256 (exited 1) Tests: 99 Failed: 8)
Failed tests: 32-39
Non-zero exit status: 1
Files=1003, Tests=27703, 50 wallclock secs ( 4.80 usr 1.51 sys + 223.02 cusr 393.93 csys = 623.26 CPU)
Result: FAIL
7950X3D (first one):
Test Summary Report
-------------------
t0050-filesystem.sh (Wstat: 256 (exited 1) Tests: 13 Failed: 0)
TODO passed: 11
Non-zero exit status: 1
t7513-interpret-trailers.sh (Wstat: 256 (exited 1) Tests: 99 Failed: 1)
Failed test: 76
Non-zero exit status: 1
Files=1003, Tests=27703, 42 wallclock secs ( 4.62 usr 1.91 sys + 238.72 cusr 653.33 csys = 898.58 CPU)
Result: FAIL
7950X3D (second one):
Test Summary Report
-------------------
t0050-filesystem.sh (Wstat: 256 (exited 1) Tests: 13 Failed: 0)
TODO passed: 11
Non-zero exit status: 1
t4101-apply-nonl.sh (Wstat: 256 (exited 1) Tests: 12 Failed: 1)
Failed test: 10
Non-zero exit status: 1
t1092-sparse-checkout-compatibility.sh (Wstat: 256 (exited 1) Tests: 103 Failed: 1)
Failed test: 9
Non-zero exit status: 1
Files=1003, Tests=27703, 42 wallclock secs ( 4.96 usr 1.87 sys + 238.87 cusr 653.84 csys = 899.54 CPU)
Result: FAIL
I'll try with some different settings (number of cores, etc.)
| 20:20:09 |
 emily | Git version bumps are not the only thing that could break Git | 20:23:07 |
| emily | passthru.tests should be for things that are particularly expensive to run, or that are very unlikely to change from other packages | 20:23:29 |
| emily | (e.g., version checks in passthru.tests are unrpboelatmci) | 20:23:45 |
| emily | * (e.g., version checks in passthru.tests are unproblematic) | 20:23:49 |
| connor (burnt/out) (UTC-8) | Interestingly enough, specifying --cores doesn't seem to affect checkPhase (but it definitely affects the build phase) of git-minimal. | 20:43:56 |
| Philip Taron (UTC-8) | Without disgreeing with this advice, they're also useful for making bootstrap easier and avoiding the sort of "just turn the damned tests off" logic that has already been applied often to the Git test suite.
If we were able to give Git's tests (which are evidently fragile flowers) their preferred operating environment it would be quite expensive.
That being said, I haven't personally worked with Git upstream and tried to make the tests more robust. | 20:56:58 |
| emily | bootstrap is the maximally load-bearing thing so I think making it easier to get through at the expense of its reliability is the wrong trade-off :) (especially since Hydra builds it even on staging unless you touch stuff that throws it away) | 20:58:00 |
| emily | it is unfortunate when we have to turn off a lot of tests though | 20:58:12 |
| emily | reliability of the end product, that is | 20:58:27 |
| emily | FWIW, doing like Hydra does and giving things few cores and scaling up jobs instead helps for a lot of flakiness. (not saying we shouldhave to do this) | 20:58:50 |
| emily | one option is turning off tests for minimal versions but not full versions | 20:59:15 |
| emily | e.g. we do that for LLVM in the Darwin bootstrap, the earlier stages do not run tests | 20:59:25 |
| Philip Taron (UTC-8) | Yes, that's exactly what I was trying to point at with "making bootstrap easier". | 20:59:49 |
| Philip Taron (UTC-8) | I completely agree. A test suite which produces false positives, even if it produces no false negatives, really impedes understanding what the reliability of the software is. | 21:01:50 |
| emily | stdenv builds fine for me on Darwin with https://github.com/NixOS/nixpkgs/pull/442167 | 23:35:16 |
| emily | so I guess it's about as ready to go as it'll ever be | 23:35:20 |
