and UNSUPPORTED"
After r292904 llvm-lit fails to emit the test results in the XML format for
Apple's internal buildbots.
rdar://30164800
llvm-svn: 292942
A `lit` condition line is now a comma-separated list of boolean expressions.
Comma-separated expressions act as if each expression were on its own
condition line:
For REQUIRES, if every expression is true then the test will run.
For UNSUPPORTED, if every expression is false then the test will run.
For XFAIL, if every expression is false then the test is expected to succeed.
As a special case "XFAIL: *" expects the test to fail.
Examples:
# Test is expected fail on 64-bit Apple simulators and pass everywhere else
XFAIL: x86_64 && apple && !macosx
# Test is unsupported on Windows and on non-Ubuntu Linux
# and supported everywhere else
UNSUPPORTED: linux && !ubuntu, system-windows
Syntax:
* '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false.
* Each test feature is a true identifier.
* Substrings of the target triple are true identifiers for UNSUPPORTED
and XFAIL, but not for REQUIRES. (This matches the current behavior.)
* All other identifiers are false.
* Identifiers are [-+=._a-zA-Z0-9]+
Differential Revision: https://reviews.llvm.org/D18185
llvm-svn: 292904
A `lit` condition line is now a comma-separated list of boolean expressions.
Comma-separated expressions act as if each expression were on its own
condition line:
For REQUIRES, if every expression is true then the test will run.
For UNSUPPORTED, if every expression is false then the test will run.
For XFAIL, if every expression is false then the test is expected to succeed.
As a special case "XFAIL: *" expects the test to fail.
Examples:
# Test is expected fail on 64-bit Apple simulators and pass everywhere else
XFAIL: x86_64 && apple && !macosx
# Test is unsupported on Windows and on non-Ubuntu Linux
# and supported everywhere else
UNSUPPORTED: linux && !ubuntu, system-windows
Syntax:
* '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false.
* Each test feature is a true identifier.
* Substrings of the target triple are true identifiers for UNSUPPORTED
and XFAIL, but not for REQUIRES. (This matches the current behavior.)
* All other identifiers are false.
* Identifiers are [-+=._a-zA-Z0-9]+
Differential Revision: https://reviews.llvm.org/D18185
llvm-svn: 292896
Running lit tests and unit tests of ASan and TSan on macOS has very bad performance when running with a high number of threads. This is caused by xnu (the macOS kernel), which currently doesn't handle mapping and unmapping of sanitizer shadow regions (reserved VM which are several terabytes large) very well. The situation is so bad that increasing the number of threads actually makes the total testing time larger. The macOS buildbots are affected by this. Note that we can't easily limit the number of sanitizer testing threads without affecting the rest of the tests.
This patch adds a special "group" into lit, and limits the number of concurrently running tests in this group. This helps solve the contention problem, while still allowing other tests to run in full, that means running lit with -j8 will still with 8 threads, and parallelism is only limited in sanitizer tests.
Differential Revision: https://reviews.llvm.org/D28420
llvm-svn: 292548
Summary:
This change equips lit.py with two new options, --num-shards=M and
--run-shard=N (set by default from env vars LIT_NUM_SHARDS and LIT_RUN_SHARD).
The options must be used together, and N must be in 1..M.
Together these options effect only test selection: they partition the testsuite
into M equal-sized "shards", then select only the Nth shard. They can be used
in a cluster of test machines to achieve a very crude (static) form of
parallelism, with minimal configuration work.
Reviewers: modocache, ddunbar
Reviewed By: ddunbar
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D28789
llvm-svn: 292417
Summary: The parameter `input` to `subprocess.Popen.communicate(...)` must be an object of type `bytes` . This is strictly enforced in python3. This patch (1) allows `to_bytes` to be safely called redundantly. (2) Explicitly convert `input` within `executeCommand`. This allows for usages like `executeCommand(['clang++', '-'], input='int main() {}\n')`.
Reviewers: ddunbar, BinaryKhaos, modocache, dim, EricWF
Reviewed By: EricWF
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D28736
llvm-svn: 292308
Running lit tests and unit tests of ASan and TSan on macOS has very bad performance when running with a high number of threads. This is caused by xnu (the macOS kernel), which currently doesn't handle mapping and unmapping of sanitizer shadow regions (reserved VM which are several terabytes large) very well. The situation is so bad that increasing the number of threads actually makes the total testing time larger. The macOS buildbots are affected by this. Note that we can't easily limit the number of sanitizer testing threads without affecting the rest of the tests.
This patch adds a special "group" into lit, and limits the number of concurrently running tests in this group. This helps solve the contention problem, while still allowing other tests to run in full, that means running lit with -j8 will still with 8 threads, and parallelism is only limited in sanitizer tests.
Differential Revision: https://reviews.llvm.org/D28420
llvm-svn: 292231
This required re-working the streaming support and lit's support for
'--gtest_list_tests' but otherwise seems to be a clean upgrade.
Differential Revision: https://reviews.llvm.org/D28154
llvm-svn: 291029
Summary:
Libc++ frequently has the need to parse more than just the builtin *test keywords* (`RUN`, `REQUIRES`, `XFAIL`, ect). For example libc++ currently needs a new keyword `MODULES-DEFINES: macro list...`. Instead of re-implementing the script parsing in libc++ this patch allows `parseIntegratedTestScript` to take custom parsers.
This patch introduces a new class `IntegratedTestKeywordParser` which implements the logic to parse/process a test keyword. Parsing of various keyword "kinds" are supported out of the box, including 'TAG', 'COMMAND', and 'LIST', which parse keywords such as `END.`, `RUN:` and `XFAIL:` respectively.
As an example after this change libc++ can implement the `MODULES-DEFINES` simply using:
```
mparser = IntegratedTestKeywordParser('MODULES-DEFINES:', ParserKind.LIST)
parseIntegratedTestScript(test, additional_parsers=[mparser])
macro_list = mparser.getValue()
```
Reviewers: ddunbar, modocache, rnk, danalbert, jroelofs
Subscribers: mgrang, llvm-commits, cfe-commits
Differential Revision: https://reviews.llvm.org/D27005
llvm-svn: 288694
This shouls now be safe and not break any more bots. It's strictly better to use '--sdk macosx', otherwise xcrun can return weird things for example when you have Command Line Tools or the SDK installed into '/'.
llvm-svn: 288385
This reverts commit r287403. It breaks an internal asan bot. According
to Kuba, a fix is up for review here: https://reviews.llvm.org/D26929
llvm-svn: 287804
Summary:
This will allow us to revert LLD r284768, which added spaces to get MSys
echo to print what we want.
Reviewers: ruiu, inglorion, rafael
Subscribers: modocache, llvm-commits
Differential Revision: https://reviews.llvm.org/D26009
llvm-svn: 285237
Summary:
r283710 introduced two regressions, one to llvm-lit, and the other to
lit executables that were installed via setuptools. Add instructions on
how to test for these regressions in the future.
Reviewers: ddunbar, delcypher, beanz, chapuni, cmatthews, echristo
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25459
llvm-svn: 284919
Update the CHECK lines in the shtest-timeout.py lit test to account for
the current output. The output has been changed in r271610 without
adjusting the tests.
Differential Revision: https://reviews.llvm.org/D25236
llvm-svn: 284057
Summary:
The Python file `utils/lit/lit/ShUtil.py` contains:
1. Logic used by lit itself
2. A set of unit tests for that logic, which can be run by invoking
`python utils/lit/lit/ShUtil.py`
Move these unit tests to a `tests/unit` subdirectory of lit, and run
the tests as part of lit's test suite. This ensures that, should the
lit test suite be included in LLVM's own regression test suite, these
unit tests will also be run.
(Instructions on how to run lit's test suite can be found in
`utils/lit/README.txt`.)
Reviewers: ddunbar, echristo, delcypher, beanz
Subscribers: mehdi_amini, llvm-commits
Differential Revision: https://reviews.llvm.org/D25411
llvm-svn: 283968
Summary:
Using Python linter flake8 on the utils/lit reveals several linter
warnings designated "F401: Unused import". Fix or silence these
warnings.
Some of these unused imports are legitimate, while some are part of lit's API.
For example, users of lit expect to be able to access `lit.formats.ShTest` in
their `lit.cfg`, despite the module hierarchy for that symbol actually being
`lit.formats.shtest.ShTest`. To silence linter errors for these lines,
include a "noqa" directive.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: mehdi_amini, llvm-commits
Differential Revision: https://reviews.llvm.org/D25407
llvm-svn: 283710
Summary:
`TestingProgressDisplay` initializes its `current` attribute to `None`, but
never reads or writes the value again. Remove it.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25415
llvm-svn: 283709
Summary:
`ArgumentError` is not defined by the Python standard library.
Executing this line of code would throw a exception, but not the
intended one. It would throw a `NameError` exception, since `ArgumentError`
is undefined.
Use `ValueError` instead, which is defined by the Python standard
library.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25410
llvm-svn: 283708
Summary:
Semicolons aren't necessary as statement terminators in Python, and
each of these uses are superfluous as they appear at the end of a line.
The convention is to not use semicolons where not needed, so remove them.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25409
llvm-svn: 283707
Summary: `prefix` is written to but never read.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25408
llvm-svn: 283706
Summary:
The minimum version of Python required to run LLVM's test suite is 2.7.
Remove a workaround for older Python versions.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25400
llvm-svn: 283705
Summary:
The minimum version of Python necessary to run the LLVM test suite is
2.7. Code to work around Python 2.5 and lower isn't necessary.
Reviewers: ddunbar, echristo, delcypher, beanz
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25209
llvm-svn: 283169
Summary:
optparse is deprecated in Python 2.7, which is the minimum version of
Python required to run the LLVM test suite. Replace its usage in lit
with argparse, optparse's 2.7 replacement module.
argparse has several benefits over optparse, but this commit does not
make use of those benefits yet. Instead, it simply uses the new API,
and attempts to keep the number of changes to a minimum.
Confirmed that lit's test suite, as well as LLVM's regression test suite,
still pass with these changes.
Patch By Brian Gesiak!
Reviewers: ddunbar, echristo, beanz, delcypher
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25173
llvm-svn: 283152
Summary:
lit's `OneCommandFileTest` class implements an abstract method that
raises if called. However, it raises by referencing an undefined
symbol. Instead, raise explicitly by throwing a `NotImplementedError`.
This is clearer, and appeases Python linters.
Patch By Brian Gesiak!
Reviewers: ddunbar, echristo, beanz
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25170
llvm-svn: 283090
Summary:
In Python, `None` is a singleton, so checking whether a variable is
`None` may be done with `is` or `is not`. This has a slight advantage
over equiality comparisons `== None` and `!= None`, since `__eq__` may
be overridden in Python to produce sometimes unexpected results.
Using `is None` and `is not None` is also recommended practice in
https://www.python.org/dev/peps/pep-0008:
> Comparisons to singletons like `None` should always be done with `is` or
> `is not`, never the equality operators.
Patch by Brian Gesiak!
Reviewers: ddunbar, echristo, beanz
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25168
llvm-svn: 283088
- This is primarily useful as a "fail fast" mode for lit, where it will stop
running tests after the first failure.
- Patch by Max Moiseev.
llvm-svn: 282452
Lots of unittests started failing under asan after r280455. It seems
they've been failing for a long time, but lit silently ignored them.
Downgrade the error so we can figure out what is going on.
Filed http://llvm.org/PR30285.
llvm-svn: 280674
Summary:
This is a follow up to r280455, where a check for the process exit code
was introduced. Some ASAN bots throw this error now, but it's impossible
to understand what's wrong with them, and the issue is not reproducible.
Reviewers: vitalybuka
Differential Revision: https://reviews.llvm.org/D24210
llvm-svn: 280550
Do this by creating a temp directory in the normal system temp
directory, and cleaning it up on exit.
It is still possible for this temp directory to leak if Python exits
abnormally, but this is probably good enough for now.
Fixes PR18335
llvm-svn: 280501
googletest formatted tests are discovered by running the test executable.
Previously testing would silently succeed if the test executable crashed
during the discovery process. Now testing fails with "error: unable to
discover google-tests ..." if the test executable exits with a non-zero status.
llvm-svn: 280455
Apparently nobody evaluated multiprocessing on Windows since Daniel
enabled multiprocessing on Unix in r193279. It works so far as I can
tell.
Today this is worth about an 8x speedup (631.29s to 73.25s) on my 24
core Windows machine. Hopefully this will improve Windows buildbot cycle
time, where currently it takes more time to run check-all than it does
to self-host with assertions enabled:
http://lab.llvm.org:8011/builders/clang-x86-windows-msvc2015/builds/20
build stage 2 ninja all ( 28 mins, 22 secs )
ninja check 2 stage 2 ( 37 mins, 38 secs )
llvm-svn: 280382
Greg Parker