There are two possible interpretations of "expected failure": either
the test *must* fail (exactly the inverse of an ordinary test, with
success becoming failure and failure becoming success), or the test
*may* fail (with success intended, but failure possible in some
environments). Autotools had the second interpretation, which seems
more useful in practice, but Meson has the first.
In GTK 3.24.35, if the environment is such that the label-sizing.ui
reftest happens to be successful, the overall result of the test suite
is failure. This seems unlikely to have been the intention.
Instead of using should_fail, put the tests in one of two new suites:
"flaky" is intended for tests that succeed or fail unpredictably
according to the test environment or chance, while "failing" is for
tests that ought to succeed but currently never do as a result of a
bug or missing functionality. With a sufficiently new version of Meson,
the flaky and failing tests are not run by default, but can be requested
with a command like:
meson test --setup=unstable_tests --suite=flaky --suite=failing
This arrangement is inspired by GNOME/glib!2987, which was contributed
by Marco Trevisan.
Signed-off-by: Simon McVittie <smcv@debian.org>
I couldn't get all reftests to work reliably, so the tests failing
on either CI or on my machine are skipped for now.
Installed tests are disabled by default and can be enabled with "-Dinstalled_tests=true"
This changes the configure option into two states:
auto: build all that can be build (default)
A list of backend names: build them and fail if we can't
"papi" is missing because it's not in Debian and I can't test it.
