Track and print the number of skipped tests. Skipped tests are tests
that should have been executed but weren't due to:
* user interrupt [Ctrl+C]
* --max-time (overall lit timeout)
* --max-failures
This is part of a larger effort to ensure that all discovered tests are
properly accounted for.
Add test for overall lit timeout feature (`--max-time` option) to
observe skipped tests. Extend test for `--max-failures` option.
Reviewed By: jdenny
Differential Revision: https://reviews.llvm.org/D77819
lit supports `--time-tests` which will report the 20 slowest tests and
print a nice histogram for test times. This change prints this list and
the histogram rows by decreasing test times. After all, we are most
interested in the slowest tests.
The LitConfig is shared across the whole test suite. However, since
enabling recursive expansion can be a breaking change for some test
suites, it's important to confine the setting to test suites that
enable it explicitly.
Note that other issues were raised with the way recursiveExpansionLimit
operates. However, this commit simply moves the setting to the right
place -- the mechanism by which it works can be improved independently.
Differential Revision: https://reviews.llvm.org/D77415
Add a new overload of StaticLibraryDefinitionGenerator::Load that takes a triple
argument and supports loading archives from MachO universal binaries in addition
to regular archives.
The LLI tool is updated to use this overload.
Graceful lit shutdown on user keyboard interrupt [Ctrl+C] was a
longstanding goal of mine. After a few refactorings this revision
finally enables it. We use the following strategy to deal with
KeyboardInterrupt:
https://noswap.com/blog/python-multiprocessing-keyboardinterrupt
Printing of a helpful summary for interrupted runs (just as the one for
completed runs) will be tackled in future revisions.
Reviewed By: serge-sans-paille, rnk
Differential Revision: https://reviews.llvm.org/D77365
On shutdown, the result complete handler is not racing with the main
thread anymore because we are now always waiting for process pool
termination via
```
finally:
pool.join()
```
The shtest-inject test relied on being executed in "single process" mode
and started to fail with a `PicklingError` after it was removed:
```
Can't pickle <class 'lit.TestingConfig.CustomFormat'>: attribute
lookup lit.TestingConfig.CustomFormat failed
```
This happened because the test config has to be serialized to the worker
process, but apparently the `CustomFormat` class defined inline is not
serializable.
This change allows passing the tested functionality (preamble_commands)
directly to `lit.formats.ShTest` so we can use it directly in the test.
Remove the "serial run" abstraction which bypasses Python's
`multiprocessing.Pool` and instead directly runs tests without spawning
worker processes. This abstraction has not offered the benefits I hoped
it would and therefore does not carry its weight.
In previous commits [1,2] I changed worker.py to only send back the test
result from the worker process instead of the whole test object. This
was a mistake. lit.Test contains fields (e.g., xfials, requires,
unsupported) that are only populated when we actually execute the test,
but are queried when we report the results in the parent process. This
commit essentially reverts the following changes:
[1] a3d2f9b53a
[2] 17bb660fb8
A previous attempt to cleanup module imports broke installing via
pip/setup.py [1]. This should be fixed now.
[1] cf252240e8
Reviewed By: paquette
Differential Revision: https://reviews.llvm.org/D76940
This minor refactoring allows reducing the amount of processing that
is duplicated when we re-run a flaky test. It also has the nice
side effect that libc++'s current test format supports flaky .sh.cpp
tests, because those are built on top of _runShTest, not executeShTest.
This allows defining substitutions in terms of other substitutions. For
example, a %build substitution could be defined in terms of a %cxx
substitution as '%cxx %s -o %t.exe' and the script would be properly
expanded.
Differential Revision: https://reviews.llvm.org/D76178
This allows creating custom test formats on top of `executeShTest` that
inject commands at the beginning of the file being parsed, without
requiring these commands to physically appear in the test file itself.
For example, one could define a test format that prints out additional
debug information at the beginning of each test. More realistically,
this has been used to define custom test formats like one that supports
compilation failure tests (e.g. with the extension `compile.fail.cpp`)
by injecting a command that calls the compiler on the file itself and
expects it to fail.
Without this change, the only alternative is to create a temporary file
with the same content as the original test, then prepend the desired
`// RUN:` lines to that file, and call `executeShTest` on that file
instead. This is both slow and cumbersome to do.
Differential Revision: https://reviews.llvm.org/D76290
This commit adds a new keyword in lit called ALLOW_RETRIES. This keyword
takes a single integer as an argument, and it allows the test to fail that
number of times before it first succeeds.
This work attempts to make the existing test_retry_attempts more flexible
by allowing by-test customization, as well as eliminate libc++'s FLAKY_TEST
custom logic.
Differential Revision: https://reviews.llvm.org/D76288
Basically change the layout to please `go build` and remove references to
`llvm-go`.
Update llvm/test/Bindings/Go/ to use the system go compiler
Differential Revision: https://reviews.llvm.org/D74540
While looking at cycle time graphs of some of my bots, I noticed
that 327894859c made check-llvm noticeably slower on macOS and
Windows.
As it turns out, the 5 substitutions added in that change were
enough to cause lit to thrash the build-in cache in re.compile()
(re.sub() is implemented as re.compile().sub()), and apparently
applySubstitutions() is on the cricital path and slow when all
regexes need to compile all the time.
(See `_MAXCACHE = 512` in cpython/Lib/re.py)
Supporting full regexes for lit substitutions seems a bit like
overkill, but for now add a simple unbounded cache to recover
the lost performance.
No intended behavior change.
A refactoring commit (cf252240) removed this line. Removing it broke installing
lit with pip and setup.py.
This adds the line back in so that we can install lit again.
For an example of how this appeared, see:
http://green.lab.llvm.org/green/job/LNT_Tests/5853/
File "/Users/buildslave/jenkins/...s/__init__.py", line 2453, in resolve
raise ImportError(str(exc))
ImportError: 'module' object has no attribute 'main'
It would appear that the removal of this lit feature was incomplete
and there is a test case that still tests for this. This patch removes
the remaining tests to bring the bots back to green. I would encourage the
author to do a post-commit review on this in case there is a more desirable fix.
Summary:
Remove REQUIRES-ANY alias lit directive since it is hardly used and can
be easily implemented using an OR expression using REQUIRES. Fixup
remaining testcases still using REQUIRES-ANY.
Reviewers: probinson, jdenny, gparker42
Reviewed By: gparker42
Subscribers: eugenis, asb, rbar, johnrusso, simoncook, sabuasal, niosHD, delcypher, jrtc27, zzheng, edward-jones, rogfer01, MartinMosbeck, brucehoult, the_o, PkmX, jocewei, lenary, s.egerton, pzheng, sameer.abuasal, apazos, luismarques, cfe-commits, #sanitizers, llvm-commits
Tags: #llvm, #clang, #sanitizers
Differential Revision: https://reviews.llvm.org/D71408
E.g. the mingw python distributed in msys2 (the mingw one, which is a
normal win32 application and doesn't use the msys2 runtime itself),
despite being a normal win32 python, still uses forward slashes. This
works fine for other cases (many, but not all), but when constructing a
raw NT path, all path separators must be backslashes.
Differential Revision: https://reviews.llvm.org/D71490
Jenkins sometimes starts a new working directory by appending @2 (or
incrementing the number if the @n suffix is already there). This causes
several clang tests to fail as:
s@INPUT_DIR@%/S/Inputs@g
gets expanded to the invalid:
s@INPUT_DIR@/path/to/workdir@2/Inputs@g
~~~~~~~~~~
where the part marked with ~'s is interpreted as the flags. These are
invalid and the test fails.
Previous fixes simply exchanged the @ character for another like | but
that's just moving the problem. Address it by adding an expansion that
escapes the @ character we're using as a delimiter as well as other magic
characters in the replacement of sed's s@@@.
There's still room for expansions to cause trouble though. One I ran into
while testing this was that having a directory called foo@bar causes lots
of `CHECK-NOT: foo` directives to match. There's also things like
directories containing `\1`
Tests go through the following stages:
*) discovered
*) filtered
*) executed
Only executed tests have a result (e.g., PASS, FAIL, XFAIL, etc.). See
"result codes" in Test.py.
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D70612
Fail early, when we discover no tests at all, or filter out all of them.
There is also `--allow-empty-runs` to disable test to allow workflows
like `LIT_FILTER=abc ninja check-all`. Apparently `check-all` invokes
lit multiple times if certain projects are enabled, which would produce
unwanted "empty runs". Specify via `LIT_OPTS=--allow-empty-runs`.
There are 3 causes for empty runs:
1) No tests discovered. This is always an error. Fix test suite config
or command line.
2) All tests filtered out. This is an error by default, but can be
suppressed via `--alow-empty-runs`. Should prevent accidentally
passing empty runs, but allow the workflow above.
3) The number of shards is greater than the number of tests. Currently,
this is never an error. Personally, I think we should consider
making this an error by default; if this happens, you are doing
something wrong. I added a warning but did not change the behavior,
since this warrants more discussion.
Reviewed By: atrick, jdenny
Differential Revision: https://reviews.llvm.org/D70105
lit's test suite calls lit multiple times for various sample test
suites. `FILECHECK_OPTS` is safe for FileCheck calls in lit's test
suite. It's not safe for FileCheck calls in the sample test suites,
whose output affects the results of lit's test suite.
Without this patch, only one such sample test suite is protected from
`FILECHECK_OPTS`, and currently `shtest-shell.py` breaks with
`FILECHECK_OPTS=-vv`. Moreover, it's hard to predict the future,
especially false passes. Thus, this patch protects all existing and
future sample test suites from `FILECHECK_OPTS` (and the deprecated
`FILECHECK_DUMP_INPUT_ON_FAILURE`).
Reviewed By: probinson
Differential Revision: https://reviews.llvm.org/D65156
Without this patch, when using lit's internal shell, if `not` on a lit
RUN line calls `env`, `diff`, or any of the other in-process shell
builtins that lit implements, lit accidentally searches for the latter
as an external executable. What's worse is that works fine when a
developer is testing on a platform where those executables are
available and behave as expected, but it then breaks on other
platforms.
`not` seems useful for some builtins, such as `diff`, so this patch
supports such uses. `not --crash` does not seem useful for builtins,
so this patch diagnoses such uses. In all cases, this patch ensures
shell builtins are found behind any sequence of `env` and `not`
commands.
`not` calling `env` calling an external command appears useful when
the `env` and external command are part of a lit substitution, as in
D65156. This patch supports that by looking through any sequence of
`env` and `not` commands, building the environment from the `env`s,
and storing the `not`s. The `not`s are then added back to the command
line without the `env`s to execute externally. This avoids the need
to replicate the `not` implementation, in particular the `--crash`
option, in lit.
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D66531
The static analyzer's scan-build script is critical infrastructure but
is not well tested. To start to address this, add a new test directory under
tests/Analysis for scan-build lit tests and seed it with several tests. The
goal is that future scan-build changes will be accompanied by corresponding
tests.
Differential Revision: https://reviews.llvm.org/D69781
Without this patch, when using lit's internal shell, if `env` on a lit
RUN line calls `env`, lit accidentally searches for the latter as an
external executable. What's worse is that works fine when a developer
is testing on a platform where `env` is available and behaves as
expected, but it then breaks on other platforms.
`env` calling `env` can make sense if one such `env` is within a lit
substitution, as in D65156 and D65121. This patch ensures that lit
executes both as internal commands.
Reviewed By: probinson, mgorny, rnk
Differential Revision: https://reviews.llvm.org/D65697
Without this patch, when using lit's internal shell, if `env` on a lit
RUN line calls `cd`, `mkdir`, or any of the other in-process shell
builtins that lit implements, lit accidentally searches for the latter
as an external executable.
This patch puts such builtins in a map so that boilerplate for them
need be implemented only once. This patch moves that handling after
processing of `env` so that `env` calling such a builtin can be
detected. Finally, because such calls appear to be useless, this
patch takes the safe approach of diagnosing them rather than
supporting them.
Reviewed By: probinson, mgorny, rnk
Differential Revision: https://reviews.llvm.org/D66506
When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:
```
# RUN: program | diff file -
```
Such cases exist now, in `clang/test/Analysis` for example. We are
preparing patches to ensure lit's internal `diff` is called in such
cases, which will then fail because lit's internal `diff` doesn't
recognize `-` as a command-line option. This patch adds support for
`-` to mean stdin.
Reviewed By: probinson, rnk
Differential Revision: https://reviews.llvm.org/D67643
When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:
```
# RUN: program | diff file -
# RUN: not diff file1 file2 | FileCheck %s
```
Such cases exist now, in `clang/test/Analysis` for example. We are
preparing patches to ensure lit's internal `diff` is called in such
cases, which will then fail because lit's internal `diff` cannot
currently be used in pipelines and doesn't recognize `-` as a
command-line option.
To enable pipelines, this patch moves lit's `diff` implementation into
an out-of-process script, similar to lit's `cat` implementation. A
follow-up patch will implement `-` to mean stdin.
Also, when lit's `diff` prints differences to stdout in Windows, this
patch ensures it always terminate lines with `\n` not `\r\n`. That
way, strict FileCheck directives checking the `diff` output succeed in
both Linux and Windows. This wasn't an issue when `diff` was internal
to lit because `diff` didn't then write to the true stdout, which is
where the `\n` -> `\r\n` conversion happened in Python.
Reviewed By: probinson, stella.stamenova
Differential Revision: https://reviews.llvm.org/D66574
Do not add user-site packages directory to the python search path.
This avoids test failures if there's an incompatible lit module installed
inside the user-site packages directory, as it gets prioritized over the lit
from the PYTHONPATH.
Large timeout values (one year, positive infinity) trip up Python on
Windows with "OverflowError: timeout value is too large". One week
seems to work and is still large enough in practice.
Thanks to Simon Pilgrim for helping me test this.
https://reviews.llvm.org/rL375171
llvm-svn: 375264
Avoid sending back the whole run.Test object (which needs to be pickled)
from the worker process when we are only interested in the test result.
llvm-svn: 375262
This will allow us to serialize just the result object instead of the
whole lit.Test object back from the worker to the main lit process.
llvm-svn: 375195
We always want to use a deadline when calling `result.await`. Let's
synthesize an artificial deadline (now plus one year) to simplify code
and do less busy waiting.
Thanks to Reid Kleckner for diagnosing that a deadline for of "positive
infinity" does not work with Python 3 anymore. See commit:
4ff1e34b60
I tested this patch with Python 2 and Python 3.
llvm-svn: 375165
Python on Windows raises this OverflowError:
gotit = waiter.acquire(True, timeout)
OverflowError: timestamp too large to convert to C _PyTime_t
So it seems this API behave the same way on every OS.
Also reverts the dependent commit a660dc590a.
llvm-svn: 375143
We always want to use a deadline when calling `result.await`. Let's
synthesize an artificial deadline (positive infinity) to simplify code
and do less busy waiting.
llvm-svn: 375129
When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:
```
# RUN: program | diff file -
```
Such cases exist now, in `clang/test/Analysis` for example. We are
preparing patches to ensure lit's internal `diff` is called in such
cases, which will then fail because lit's internal `diff` doesn't
recognize `-` as a command-line option. This patch adds support for
`-` to mean stdin.
Reviewed By: probinson, rnk
Differential Revision: https://reviews.llvm.org/D67643
llvm-svn: 375116
When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:
```
# RUN: program | diff file -
# RUN: not diff file1 file2 | FileCheck %s
```
Such cases exist now, in `clang/test/Analysis` for example. We are
preparing patches to ensure lit's internal `diff` is called in such
cases, which will then fail because lit's internal `diff` cannot
currently be used in pipelines and doesn't recognize `-` as a
command-line option.
To enable pipelines, this patch moves lit's `diff` implementation into
an out-of-process script, similar to lit's `cat` implementation. A
follow-up patch will implement `-` to mean stdin.
Reviewed By: probinson, stella.stamenova
Differential Revision: https://reviews.llvm.org/D66574
llvm-svn: 375114
* Remove outdated precautions for Python versions < 2.7
* Remove dead code related to `maxIndividualTestTime` option
* Move printing of test and result summary out of main into its own
function
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D68847
llvm-svn: 375046
Julian Lettner