Currently, a LIT test named directly (on the command line) will
be run even if the name of the test file does not meet the rules
to be considered a test in the LIT test configuration files for
its test suite. For example, if the test does not have a
recognised file extension.
This makes it relatively easy to write a LIT test that won't
actually be run. I did in: https://reviews.llvm.org/D82567
This patch adds an error to avoid users doing that. There is a
small performance overhead for this check. A command line option
has been added so that users can opt into the old behaviour.
Differential Revision: https://reviews.llvm.org/D83069
The tests previously relied on the `short.py` and `FirstTest.subTestA`
script being executed on a machine within a short time window (1 or 2
seconds). While this "seems to work" it can fail on resource constrained
machines. We could bump the timeout a little bit (bumping it too
much would mean the test would take a long time to execute) but it wouldn't
really solve the problem of the test being prone to failures.
This patch tries to remove this flakeyness by separating testing into
two separate parts:
1. Testing if a test can hit a timeout.
2. Testing if a test can run to completion in the presence of a
timeout.
This way we can give (1.) a really short timeout (to make the test run
as fast as possible) and (2.) a really long timeout. This means for (2.)
we are no longer trying to rely on the "short" test executing within
some short time window. Instead the window is now 3600 seconds which
should be long enough even for a heavily resource constrained machine to
execute the "short" test.
Thanks to Julian Lettner for suggesting this approach. This superseeds
my original approach in https://reviews.llvm.org/D88807.
This patch also changes the command line override test to run the quick
test rather than the slow one to make the test run faster.
Differential Revision: https://reviews.llvm.org/D89020
Failing test output sometimes contains control characters like \x1b (e.g.
if there was some -fcolor-diagnostics output) which are not allowed inside
XML files. This causes problems with CI systems: for example, the Jenkins
JUnit XML will throw an exception when ecountering those characters and
similar problems also occur with GitLab CI.
Reviewed By: yln, jdenny
Differential Revision: https://reviews.llvm.org/D84233
Otherwise, if a Lit script contains escaped substitutions (like %%p in this test https://github.com/llvm/llvm-project/blob/master/compiler-rt/test/asan/TestCases/Darwin/asan-symbolize-partial-report-with-module-map.cpp#L10), they are unescaped during recursive application of substitutions, and the results are unexpected.
We solve it using the fact that double percent signs are first replaced with #_MARKER_#, and only after all the other substitutions have been applied, #_MARKER_# is replaced with a single percent sign. The only change is that instead of replacing #_MARKER_# at each recursion step, we replace it once after the last recursion step.
Differential Revision: https://reviews.llvm.org/D83894
When running multiple shards, don't include skipped tests in the xunit
output since merging the files will result in duplicates.
In our CHERI Jenkins CI, I configured the libc++ tests to run using sharding
(since we are testing using a single-CPU QEMU). We then merge the generated
XUnit xml files to produce a final result, but if the individual XMLs
report tests excluded due to sharding each test is included N times in the
final result. This also makes it difficult to find the tests that were
skipped due to missing REQUIRES: etc.
Reviewed By: yln
Differential Revision: https://reviews.llvm.org/D84235
As per discussion in D69207, have lit ignore UnicodeDecodeErrors
when running with python 2 in an ASCII shell.
Differential Revision: https://reviews.llvm.org/D82754
Provide `--show-xxx` flags for all non-failure result codes, just as we
already do for `--show-xfail` and `--show-unsupported`.
Reviewed By: jdenny
Differential Revision: https://reviews.llvm.org/D82233
Summary: This is a complementary patch to D82100 since the aix builbot is still running the unsupported test shtest-format-argv0. Add system-aix to the sub llvm-lit config.
Reviewers: daltenty, hubert.reinterpretcast
Reviewed By: hubert.reinterpretcast
Subscribers: delcypher, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D82905
Summary: lit test `shtest-format.py` fails on AIX because one of the subtest of shtest-format requires the tool `[` to be installed under the system PATH. For AIX, `[` is only available as a shell builtin and does not present as an executable file under PATH. Hence, split the original shtest-format into two separate test files and added AIX as UNSUPPORTED for the test using `[` .
Reviewers: daltenty, hubert.reinterpretcast
Reviewed By: hubert.reinterpretcast
Subscribers: delcypher, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D82100
Let's have one canonical place to define ResultCode instances and their
labels.
Also make ResultCode's `__init__` function self-registering to better
support custom ResultCodes.
Before this change we showed all result groups with a code that was not
explicitly hard-coded set. This set missed the FLAKYPASS result code.
Let's generalize the code to always show failures and the additionally
requested result codes.
MSVC uses lit for STL testing to run both the libcxx tests and our "native" suite of tests which has feature requirements that are not parsed from the test content. For consistency, the change treats the `unsupported` and `xfails` `Test` properties similarly to `requires`.
Differential Revision: https://reviews.llvm.org/D81782
Pass in all discovered tests to report generators.
The XunitReport generator now creates testcase items for unexecuted
tests and documents why they have been skipped. This makes it easier
to compare test runs with different filters or configurations, or across
platforms.
I don't know who is using the JsonReport generator and what the
expectations there are (it doesn't have tests), so decided to preserve
the old behavior by filtering out the unexecuted tests.
Reviewed By: jdenny
Differential Revision: https://reviews.llvm.org/D81316
In TestRunner.py, D78589 extracts a `_parseKeywords` function from
`parseIntegratedTestScript`, which then expects `_parseKeywords` to
always return a list of keyword/value pairs. However, the extracted
code sometimes returns an unresolved `lit.Test.Result` on a keyword
parsing error, which then produces a stack dump instead of the
expected diagnostic.
This patch fixes that, makes the style of those diagnostics more
consistent, and extends the lit test suite to cover them.
Reviewed By: ldionne
Differential Revision: https://reviews.llvm.org/D81665
Having the input dumped on failure seems like a better
default: I debugged FileCheck tests for a while without knowing
about this option, which really helps to understand failures.
Remove `-dump-input-on-failure` and the environment variable
FILECHECK_DUMP_INPUT_ON_FAILURE which are now obsolete.
Differential Revision: https://reviews.llvm.org/D81422
Improve consistency when printing test results:
Previously we were using different labels for group names (the header
for the list of, e.g., failing tests) and summary count lines. For
example, "Failing Tests"/"Unexpected Failures". This commit changes lit
to label things consistently.
Improve wording of labels:
When talking about individual test results, the first word in
"Unexpected Failures", "Expected Passes", and "Individual Timeouts" is
superfluous. Some labels contain the word "Tests" and some don't.
Let's simplify the names.
Before:
```
Failing Tests (1):
...
Expected Passes : 3
Unexpected Failures: 1
```
After:
```
Failed Tests (1):
...
Passed: 3
Failed: 1
```
Reviewed By: ldionne
Differential Revision: https://reviews.llvm.org/D77708
Lit test suites can tend to accumulate annotations that are not necessarily
relevant as time goes by, for example XFAILS on old compilers or platforms.
To help spot old annotations that can be cleaned up, it can be useful to
look at all features used inside a test suite.
This commit adds a new Lit option '--show-used-features' that prints all
the features used in XFAIL, REQUIRES and UNSUPPORTED of all tests that
are discovered.
Differential Revision: https://reviews.llvm.org/D78589
This reverts commit 78dea0e8fb.
The offending lldb test (which is a real bug exposed by this patch)
has been disabled on windows (see a67b2faa7c)
and lldb is queued for inclusion into precommit testing, which would
have caught this.
Differential Revision: https://reviews.llvm.org/D80389
lit runs a gtest executable multiple times. First it runs it to
discover tests, then later it runs the executable again for each test.
However, if the discovery fails (perhaps because of a broken
executable), then no tests were previously run and no failures were
reported. This patch creates a dummy test if discovery fails, which
will later fail when test are run and be reported as a failure.
Differential Revision: https://reviews.llvm.org/D80096
abhinavgaba reported that that the custom-result-category.py test hangs
on a Windows build bot [1]. Disable it for now.
[1] https://reviews.llvm.org/D78164#2018178
Track and print the number of tests that were discovered but not
executed due to test selection options:
* --filter (regex filter)
* --max-tests (limits number of tests)
* sharding feature
With this change all discovered tests are accounted for: every
discovered test is included in one of the counts printed in the summary.
Reviewed By: jdenny
Differential Revision: https://reviews.llvm.org/D78078
The lnt test suite defines custom result codes [1]. Support those via
an extension API instead of "by accident", which should offer the
advantage of properly handling them when we print test results.
[1] https://reviews.llvm.org/D77986
Reviewed By: Meinersbur
Differential Revision: https://reviews.llvm.org/D78164
This reverts commit bc3f54de18.
The patch breaks in the following two scenarios:
1. When manually passing an absolute path to llvm-lit with a lower-case
drive letter: `python bin\llvm-lit.py -sv c:\llvm-project\clang\test\PCH`
2. When the PWD has a lower-case drive letter, like after running
`cd c:\` with a lower-case "c:" (cmd's default is upper-case, but
it takes case-ness from what's passed to `cd` apparently).
There's been some back and forth if the cfg paths in the
config_map should be normcase()d. The argument for is that
it allows using all-lower spelling in cmd on Windows, the
argument against that doing so is lossy.
Before the relative-paths-in-generated-lit.site.cfg.py work,
there was no downside to calling normcase(), but with it
we need a hack to recover the original case.
This time, normcase() the hashtable key, but store the original
cased key in addition to the value. This fixes both cons, at the
cost of a few bytes more memory.
Differential Revision: https://reviews.llvm.org/D78169
We already print available features, and it can be useful to print
substitutions as well since those are a pretty fundamental part of
a test suite. We could also consider printing other things like the
test environment, however the need doesn't appear to be as strong.
As a fly-by fix, we also always print available features, even when
there are none.
Before:
$ lit -sv libcxx/test --show-suites
-- Test Suites --
libc++ - 6350 tests
Source Root: [...]
Exec Root : [...]
Available Features : -faligned-allocation -fsized-deallocation [...]
After:
$ lit -sv libcxx/test --show-suites
-- Test Suites --
libc++ - 6350 tests
Source Root: [...]
Exec Root : [...]
Available Features: -faligned-allocation -fsized-deallocation [...]
Available Substitutions: %{build_module} => [...]
%{build} => %{cxx} -o [...]
Differential Revision: https://reviews.llvm.org/D77818
max-time.py:
Windows does not have a native `sleep` command, use `time.sleep()` in
Python instead.
max-failures.py:
The max-failure test reused the shtest-shell test inputs instead of
defining its own "test domain". However, the output of this
shtest-shell "test domain" is slightly different on Windows, which now
bites us since we made the max-failures test stricter. Let's define
our own "max failures" test domain.
Fixup for cbe42a9d5f. Increase values for testing the overall lit
timeout (--max-time) which wasn't enough for the test to complete on
very slow build bots.
Track and print the number of skipped tests. Skipped tests are tests
that should have been executed but weren't due to:
* user interrupt [Ctrl+C]
* --max-time (overall lit timeout)
* --max-failures
This is part of a larger effort to ensure that all discovered tests are
properly accounted for.
Add test for overall lit timeout feature (`--max-time` option) to
observe skipped tests. Extend test for `--max-failures` option.
Reviewed By: jdenny
Differential Revision: https://reviews.llvm.org/D77819
The LitConfig is shared across the whole test suite. However, since
enabling recursive expansion can be a breaking change for some test
suites, it's important to confine the setting to test suites that
enable it explicitly.
Note that other issues were raised with the way recursiveExpansionLimit
operates. However, this commit simply moves the setting to the right
place -- the mechanism by which it works can be improved independently.
Differential Revision: https://reviews.llvm.org/D77415
The shtest-inject test relied on being executed in "single process" mode
and started to fail with a `PicklingError` after it was removed:
```
Can't pickle <class 'lit.TestingConfig.CustomFormat'>: attribute
lookup lit.TestingConfig.CustomFormat failed
```
This happened because the test config has to be serialized to the worker
process, but apparently the `CustomFormat` class defined inline is not
serializable.
This change allows passing the tested functionality (preamble_commands)
directly to `lit.formats.ShTest` so we can use it directly in the test.
This allows defining substitutions in terms of other substitutions. For
example, a %build substitution could be defined in terms of a %cxx
substitution as '%cxx %s -o %t.exe' and the script would be properly
expanded.
Differential Revision: https://reviews.llvm.org/D76178
This allows creating custom test formats on top of `executeShTest` that
inject commands at the beginning of the file being parsed, without
requiring these commands to physically appear in the test file itself.
For example, one could define a test format that prints out additional
debug information at the beginning of each test. More realistically,
this has been used to define custom test formats like one that supports
compilation failure tests (e.g. with the extension `compile.fail.cpp`)
by injecting a command that calls the compiler on the file itself and
expects it to fail.
Without this change, the only alternative is to create a temporary file
with the same content as the original test, then prepend the desired
`// RUN:` lines to that file, and call `executeShTest` on that file
instead. This is both slow and cumbersome to do.
Differential Revision: https://reviews.llvm.org/D76290
This commit adds a new keyword in lit called ALLOW_RETRIES. This keyword
takes a single integer as an argument, and it allows the test to fail that
number of times before it first succeeds.
This work attempts to make the existing test_retry_attempts more flexible
by allowing by-test customization, as well as eliminate libc++'s FLAKY_TEST
custom logic.
Differential Revision: https://reviews.llvm.org/D76288
It would appear that the removal of this lit feature was incomplete
and there is a test case that still tests for this. This patch removes
the remaining tests to bring the bots back to green. I would encourage the
author to do a post-commit review on this in case there is a more desirable fix.
Summary:
Remove REQUIRES-ANY alias lit directive since it is hardly used and can
be easily implemented using an OR expression using REQUIRES. Fixup
remaining testcases still using REQUIRES-ANY.
Reviewers: probinson, jdenny, gparker42
Reviewed By: gparker42
Subscribers: eugenis, asb, rbar, johnrusso, simoncook, sabuasal, niosHD, delcypher, jrtc27, zzheng, edward-jones, rogfer01, MartinMosbeck, brucehoult, the_o, PkmX, jocewei, lenary, s.egerton, pzheng, sameer.abuasal, apazos, luismarques, cfe-commits, #sanitizers, llvm-commits
Tags: #llvm, #clang, #sanitizers
Differential Revision: https://reviews.llvm.org/D71408
Ben Dunbobbin