Running lit tests and unit tests of ASan and TSan on macOS has very bad performance when running with a high number of threads. This is caused by xnu (the macOS kernel), which currently doesn't handle mapping and unmapping of sanitizer shadow regions (reserved VM which are several terabytes large) very well. The situation is so bad that increasing the number of threads actually makes the total testing time larger. The macOS buildbots are affected by this. Note that we can't easily limit the number of sanitizer testing threads without affecting the rest of the tests.
This patch adds a special "group" into lit, and limits the number of concurrently running tests in this group. This helps solve the contention problem, while still allowing other tests to run in full, that means running lit with -j8 will still with 8 threads, and parallelism is only limited in sanitizer tests.
Differential Revision: https://reviews.llvm.org/D28420
llvm-svn: 292231
This required re-working the streaming support and lit's support for
'--gtest_list_tests' but otherwise seems to be a clean upgrade.
Differential Revision: https://reviews.llvm.org/D28154
llvm-svn: 291029
Summary:
Libc++ frequently has the need to parse more than just the builtin *test keywords* (`RUN`, `REQUIRES`, `XFAIL`, ect). For example libc++ currently needs a new keyword `MODULES-DEFINES: macro list...`. Instead of re-implementing the script parsing in libc++ this patch allows `parseIntegratedTestScript` to take custom parsers.
This patch introduces a new class `IntegratedTestKeywordParser` which implements the logic to parse/process a test keyword. Parsing of various keyword "kinds" are supported out of the box, including 'TAG', 'COMMAND', and 'LIST', which parse keywords such as `END.`, `RUN:` and `XFAIL:` respectively.
As an example after this change libc++ can implement the `MODULES-DEFINES` simply using:
```
mparser = IntegratedTestKeywordParser('MODULES-DEFINES:', ParserKind.LIST)
parseIntegratedTestScript(test, additional_parsers=[mparser])
macro_list = mparser.getValue()
```
Reviewers: ddunbar, modocache, rnk, danalbert, jroelofs
Subscribers: mgrang, llvm-commits, cfe-commits
Differential Revision: https://reviews.llvm.org/D27005
llvm-svn: 288694
This shouls now be safe and not break any more bots. It's strictly better to use '--sdk macosx', otherwise xcrun can return weird things for example when you have Command Line Tools or the SDK installed into '/'.
llvm-svn: 288385
This reverts commit r287403. It breaks an internal asan bot. According
to Kuba, a fix is up for review here: https://reviews.llvm.org/D26929
llvm-svn: 287804
Summary:
This will allow us to revert LLD r284768, which added spaces to get MSys
echo to print what we want.
Reviewers: ruiu, inglorion, rafael
Subscribers: modocache, llvm-commits
Differential Revision: https://reviews.llvm.org/D26009
llvm-svn: 285237
Summary:
r283710 introduced two regressions, one to llvm-lit, and the other to
lit executables that were installed via setuptools. Add instructions on
how to test for these regressions in the future.
Reviewers: ddunbar, delcypher, beanz, chapuni, cmatthews, echristo
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25459
llvm-svn: 284919
Update the CHECK lines in the shtest-timeout.py lit test to account for
the current output. The output has been changed in r271610 without
adjusting the tests.
Differential Revision: https://reviews.llvm.org/D25236
llvm-svn: 284057
Summary:
The Python file `utils/lit/lit/ShUtil.py` contains:
1. Logic used by lit itself
2. A set of unit tests for that logic, which can be run by invoking
`python utils/lit/lit/ShUtil.py`
Move these unit tests to a `tests/unit` subdirectory of lit, and run
the tests as part of lit's test suite. This ensures that, should the
lit test suite be included in LLVM's own regression test suite, these
unit tests will also be run.
(Instructions on how to run lit's test suite can be found in
`utils/lit/README.txt`.)
Reviewers: ddunbar, echristo, delcypher, beanz
Subscribers: mehdi_amini, llvm-commits
Differential Revision: https://reviews.llvm.org/D25411
llvm-svn: 283968
Summary:
Using Python linter flake8 on the utils/lit reveals several linter
warnings designated "F401: Unused import". Fix or silence these
warnings.
Some of these unused imports are legitimate, while some are part of lit's API.
For example, users of lit expect to be able to access `lit.formats.ShTest` in
their `lit.cfg`, despite the module hierarchy for that symbol actually being
`lit.formats.shtest.ShTest`. To silence linter errors for these lines,
include a "noqa" directive.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: mehdi_amini, llvm-commits
Differential Revision: https://reviews.llvm.org/D25407
llvm-svn: 283710
Summary:
`TestingProgressDisplay` initializes its `current` attribute to `None`, but
never reads or writes the value again. Remove it.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25415
llvm-svn: 283709
Summary:
`ArgumentError` is not defined by the Python standard library.
Executing this line of code would throw a exception, but not the
intended one. It would throw a `NameError` exception, since `ArgumentError`
is undefined.
Use `ValueError` instead, which is defined by the Python standard
library.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25410
llvm-svn: 283708
Summary:
Semicolons aren't necessary as statement terminators in Python, and
each of these uses are superfluous as they appear at the end of a line.
The convention is to not use semicolons where not needed, so remove them.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25409
llvm-svn: 283707
Summary: `prefix` is written to but never read.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25408
llvm-svn: 283706
Summary:
The minimum version of Python required to run LLVM's test suite is 2.7.
Remove a workaround for older Python versions.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25400
llvm-svn: 283705
Summary:
The minimum version of Python necessary to run the LLVM test suite is
2.7. Code to work around Python 2.5 and lower isn't necessary.
Reviewers: ddunbar, echristo, delcypher, beanz
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25209
llvm-svn: 283169
Summary:
optparse is deprecated in Python 2.7, which is the minimum version of
Python required to run the LLVM test suite. Replace its usage in lit
with argparse, optparse's 2.7 replacement module.
argparse has several benefits over optparse, but this commit does not
make use of those benefits yet. Instead, it simply uses the new API,
and attempts to keep the number of changes to a minimum.
Confirmed that lit's test suite, as well as LLVM's regression test suite,
still pass with these changes.
Patch By Brian Gesiak!
Reviewers: ddunbar, echristo, beanz, delcypher
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25173
llvm-svn: 283152
Summary:
lit's `OneCommandFileTest` class implements an abstract method that
raises if called. However, it raises by referencing an undefined
symbol. Instead, raise explicitly by throwing a `NotImplementedError`.
This is clearer, and appeases Python linters.
Patch By Brian Gesiak!
Reviewers: ddunbar, echristo, beanz
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25170
llvm-svn: 283090
Summary:
In Python, `None` is a singleton, so checking whether a variable is
`None` may be done with `is` or `is not`. This has a slight advantage
over equiality comparisons `== None` and `!= None`, since `__eq__` may
be overridden in Python to produce sometimes unexpected results.
Using `is None` and `is not None` is also recommended practice in
https://www.python.org/dev/peps/pep-0008:
> Comparisons to singletons like `None` should always be done with `is` or
> `is not`, never the equality operators.
Patch by Brian Gesiak!
Reviewers: ddunbar, echristo, beanz
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25168
llvm-svn: 283088
- This is primarily useful as a "fail fast" mode for lit, where it will stop
running tests after the first failure.
- Patch by Max Moiseev.
llvm-svn: 282452
Lots of unittests started failing under asan after r280455. It seems
they've been failing for a long time, but lit silently ignored them.
Downgrade the error so we can figure out what is going on.
Filed http://llvm.org/PR30285.
llvm-svn: 280674
Summary:
This is a follow up to r280455, where a check for the process exit code
was introduced. Some ASAN bots throw this error now, but it's impossible
to understand what's wrong with them, and the issue is not reproducible.
Reviewers: vitalybuka
Differential Revision: https://reviews.llvm.org/D24210
llvm-svn: 280550
Do this by creating a temp directory in the normal system temp
directory, and cleaning it up on exit.
It is still possible for this temp directory to leak if Python exits
abnormally, but this is probably good enough for now.
Fixes PR18335
llvm-svn: 280501
googletest formatted tests are discovered by running the test executable.
Previously testing would silently succeed if the test executable crashed
during the discovery process. Now testing fails with "error: unable to
discover google-tests ..." if the test executable exits with a non-zero status.
llvm-svn: 280455
Apparently nobody evaluated multiprocessing on Windows since Daniel
enabled multiprocessing on Unix in r193279. It works so far as I can
tell.
Today this is worth about an 8x speedup (631.29s to 73.25s) on my 24
core Windows machine. Hopefully this will improve Windows buildbot cycle
time, where currently it takes more time to run check-all than it does
to self-host with assertions enabled:
http://lab.llvm.org:8011/builders/clang-x86-windows-msvc2015/builds/20
build stage 2 ninja all ( 28 mins, 22 secs )
ninja check 2 stage 2 ( 37 mins, 38 secs )
llvm-svn: 280382
This variable is used by ASan (and other sanitizers in the future)
on s390x-linux to override a check for CVE-2016-2143 in the running
kernel (see revision 267747 on compiler-rt). Since the check simply
checks if the kernel version is in a whitelist of known-good versions,
it may miss distribution kernels, or manually-patched kernels - hence
the need for this variable. To enable running the ASan testsuite on
such kernels, this variable should be passed from the environment
down to the testcases.
Differential Revision: http://reviews.llvm.org/D19888
llvm-svn: 273825
- The intended use of this was just in diagnostics, so we shouldn't pay the
cost of reading these all the time.
- This will avoid including the full output of each command in tests which
fail, but the most important use case for this was to gather the output of
the specific command which failed.
llvm-svn: 272365
Use os.devnull instead of tempfiles when substituting '/dev/null' on
Windows machines. This should make the bots just a bit speedier.
Thanks to Yunzhong Gao for testing this patch on Windows!
Differential Revision: http://reviews.llvm.org/D20549
llvm-svn: 272290
- This will cause lit to automatically include the first 1K of data in
redirected output files when a command fails (previously if the command
failed, but the main point of the test was, say, a `FileCheck` later on, then
the log wasn't helpful in showing why the command failed).
llvm-svn: 272021
- This only applies to scripts executed by the _internal_ shell script
interpreter.
- This patch reworks the log to look more like a shell transcript, and be less
verbose (but in the interest of calling attention to the important parts).
Here is an example of the new format, for commands with/without failures and
with/without output:
```
$ true
$ echo hi
hi
$ false
note: command had no output on stdout or stderr
error: command failed with exit status 1
```
llvm-svn: 271610
Summary:
This patch adds a "REQUIRES-ANY" feature test that is disjunctive. This marks a test as `UNSUPPORTED` if none of the specified features are available.
Libc++ has the need to write feature test such as `// REQUIRES-ANY: c++98, c++03` when testing of behavior that is specific to older dialects but has since changed.
Reviewers: rnk, ddunbar
Subscribers: ddunbar, probinson, llvm-commits, cfe-commits
Differential Revision: http://reviews.llvm.org/D20757
llvm-svn: 271468
Summary:
We need these variables to concatenate two absolute paths to construct
a valid path. Currently, %t\%t is, for example, expanded to C:\foo\C:\foo,
which is not a valid path because ":" is not a valid path character
on Windows. With this patch, %t will be expanded to C\foo.
Differential Revision: http://reviews.llvm.org/D19757
llvm-svn: 268168
Summary:
Upstream googletest prints "Running main() from gtest_main.cc" to stdout prior
to running tests. LLVM removed that print statement in r61540. If a user were
to use lit to run tests that use upstream googletest, however, lit
reports "Running main()" as an invalid test name.
To avoid such a failure, add an extra conditional to `formats/googletest.py`.
Also add tests to demonstrate the modified behavior.
Reviewers: abdulras, ddunbar
Subscribers: ddunbar, llvm-commits, kastiglione
Differential Revision: http://reviews.llvm.org/D18606
llvm-svn: 265034
Summary:
The multiprocessing.Queue.put() call can hang if we try queueing all the
tests before starting to take them out of the queue.
The current implementation hangs if tests exceed 2^^15, on Mac OS X.
This might happen with a ninja check-all if one has a bunch of llvm
projects.
Reviewers: delcypher, bkramer
Subscribers: llvm-commits
Differential Revision: http://reviews.llvm.org/D17609
llvm-svn: 263731
This lets us for example start running the unit test suite early. For
'check-llvm' on my machine, this drops the tim e from 44s to 32s!!!!!
It's pretty ugly. I barely know how to write Python, so feel free to
just tell me how I should write it instead. =D Thanks to Filipe and
others for help.
Differential Revision: http://reviews.llvm.org/D18089
llvm-svn: 263329
Fix the lit bug that enabled this "feature" (empty triple is substring
of all possible target triples) and change the two outliers to use the
documented * syntax.
llvm-svn: 259799
If a lit test has a RUN line that includes a redirection to "/dev/tty", the
redirection goes to the special device file corresponding to the console. It
is /dev/tty on UNIX-like systems and "CON" on Windows.
This patch is needed to implement a test like PR25717 (caused by the size limit
of the Windows system call WriteConsole() prior to Windows 8) where the test
only breaks when outputing to the console and won't fail if using a pipe.
llvm-svn: 258898
is < ``2.0``.
Older versions of psutil (e.g. ``1.2.1`` which is the version shipped with
Ubuntu 14.04) use a different API for retrieving the child processes.
To handle this try the new API first and if that fails try the old API.
llvm-svn: 257616
This reverts r257221.
This caused several build bot failures
* It looks like some of the tests don't work correctly under Windows
* It looks like the lit per test timeout tests fail
So I'm reverting for now. Once the above failures are fixed running
lit's tests can be enabled again.
llvm-svn: 257268
directy with ``make check-lit`` and are run as part of
``make check-all``.
In principle we should run lit's testsuite before testing LLVM using lit
so that any problems with lit get discovered before testing LLVM so we
can bail out early. However this implementation (``check-all`` runs all
tests together) seemed simpler and will still report failing lit tests.
Note that the tests and the configured ``lit.site.cfg`` have to be
copied into the build directory to avoid polluting the source tree.
llvm-svn: 257221
This should work with ShTest (executed externally or internally) and GTest
test formats.
To set the timeout a new option ``--timeout=`` has
been added which specifies the maximum run time of an individual test
in seconds. By default this 0 which causes no timeout to be enforced.
The timeout can also be set from a lit configuration file by modifying
the ``lit_config.maxIndividualTestTime`` property.
To implement a timeout we now require the psutil Python module if a
timeout is requested. This dependency is confined to the newly added
``lit.util.killProcessAndChildren()``. A note has been added into the
TODO document describing how we can remove the dependency on the
``pustil`` module in the future. It would be nice to remove this
immediately but that is a lot more work and Daniel Dunbar believes it is
better that we get a working implementation first and then improve it.
To avoid breaking the existing behaviour the psutil module will not be
imported if no timeout is requested.
The included testcases are derived from test cases provided by
Jonathan Roelofs which were in an previous attempt to add a per test
timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan!
Reviewers: ddunbar, jroelofs, cmatthews, MatzeB
Subscribers: cmatthews, llvm-commits
Differential Revision: http://reviews.llvm.org/D14706
llvm-svn: 256471
the script when running a ShTest with an external or internal shell.
This bug is caused by use of the ``map`` function in Python 3 which
returns an iterable (rather than a list in Python 2). After the iterable
is exhausted it won't return any more output and consequently when
``_runShTest()`` tries to access the ``script`` which has already been
iterated over it is empty. Converting to a list immediatley after
calling ``map()`` fixes this.
This fixes the ``tests/shtest-format.py`` test when running under
Python3 which was previously failing.
llvm-svn: 253556
instead of executable if the argument was found inside a directory
contained in PATH.
An example where this could cause a problem is if there was a RUN line
that ran the ``test`` command and if the user had a directory in their
PATH that contained a directory called ``test/`` (that occured before
``/usr/bin/``). Lit would try to use the directory as the executable
which would fail with the rather cryptic message.
```
Could not create process due to [Errno 13] Permission denied
```
llvm-svn: 253031
The existing -v option only displays commands and outputs for failed
tests, the newly introduced -a displays it for all executed tests.
llvm-svn: 251806
It is common to have a default soft process limit, at least on some families of
Linux distributions, of 1024. This is normally more than enough, but if you
have many cores, and you're running tests that create many threads, this can
become a problem. My POWER7 development machine has 48 cores, and when running
the lld regression tests, which often want to create up to 48 threads, I run
into problems. lit, by default, will want to run 48 tests in parallel, and
48*48 < 1024, and so many tests fail like this:
terminate called after throwing an instance of 'std::system_error'
what(): Resource temporarily unavailable
or lit fails like this when launching a test:
OSError: [Errno 11] Resource temporarily unavailable
lit can easily detect this situation and attempt to repair it before launching
tests (by raising the soft process limit to something that will allow ncpus^2
threads to be created), and should do so to prevent spurious test failures.
This is the follow-up to this thread:
http://lists.llvm.org/pipermail/llvm-dev/2015-October/090942.html
llvm-svn: 249161
The variable is actually called ANDROID_SERIAL.
This was not exercised on the bots until today.
Should fix the sanitizer-x86_64-linux failures.
llvm-svn: 246898
The plan is to use this for the sanitizer test suite on Windows. See
PR24554 for more details on why we need this.
Tested manually by injecting rand() into a sanitizer test and watching
what it does.
llvm-svn: 246704
My theory is that somehow Python's refcounting and GC strategy isn't
closing the subprocess handle in a timely fashion. This accesses the
private '_handle' field of the Popen object, but I see no other way to
do this. If this doesn't address the problem on the sanitizer-windows
buildbot, we can revert this change. If it does, then let's keep the
hack.
llvm-svn: 245946
This was affecting test/asan/TestCases/Windows/coverage-basic.cc in
compiler-rt. It does something like:
cd %T/mydir
%clang %s -o t.exe
./t.exe
Previously, we'd end up looking for t.exe relative to the cwd of the lit
process, not the cwd of the test.
llvm-svn: 242941
The MSys 2 version of 'env' cannot be used to set 'TZ' in the
environment due to some portability hacks in the process spawning
compatibility layer[1]. This affects test/Object/archive-toc.test, which
tries to set TZ in the environment.
Other than that, this saves a subprocess invocation of a small unix
utility, which is makes the tests faster.
The internal shell does not support shell variable expansion, so this
idiom in the ASan tests isn't supported yet:
RUN: env ASAN_OPTIONS=$ASAN_OPTIONS:opt=1 ...
[1] https://github.com/Alexpux/MSYS2-packages/issues/294
Differential Revision: http://reviews.llvm.org/D11350
llvm-svn: 242696
Summary: This patch allows executeCommand to pass a string to the processes stdin.
Reviewers: ddunbar, jroelofs
Subscribers: llvm-commits
Differential Revision: http://reviews.llvm.org/D11332
llvm-svn: 242631
Pass ADB and ADB_SERIAL environment variables to lit tests.
This would allow running Android tests in compiler-rt when
there is more than one device attached to the host.
llvm-svn: 240459
Summary:
I spend some time trying to get the LIT test suite passing. Here are the changes that I needed to make on my machine.
I made the following changes for the following reasons.
1. google-test.py: The Google test format now checks for "[ PASSED ] 1 test." to check if a test passes.
2. discovery.py: The output appears in a different order on my machine than it did in the test.
3. unittest-adaptor.py: The output appears in a different order on my machine than it did in the test.
4. The classname is now formed differently in `getJUnitXML(...)`.
I'm not sure what is causing the output order to differ in discovery.py and unittest-adaptor.py. Does anybody have any thoughts?
Reviewers: ddunbar, danalbert, jroelofs
Reviewed By: jroelofs
Subscribers: llvm-commits
Differential Revision: http://reviews.llvm.org/D9864
llvm-svn: 239663
By setting limit_to_features to a non empty list of features a configuration can
restrict the set of tests to run to only include tests that require a feature in
this list.
rdar://21082253
llvm-svn: 238766
These changes allow usages where you want to pass an additional
commandline option to all invocations of a specific llvm tool. Example:
> llvm-lit -Dllc=llc -enable-misched -verify-machineinstrs
Differential Revision: http://reviews.llvm.org/D9487
llvm-svn: 236461
The sanitizer test suite uses this idiom to disable a test. Now that we
actually check if a test ran after invoking it, we see that zero tests
ran, and complain.
Instead, ignore tests starting with DISABLED_ completely. Fixes the
sanitizer test suite failures on Windows.
llvm-svn: 234247
The '/' character in the test name of a type-parameterized test is not a
path separator, and should not be '\' on Windows. We were passing a test
name to --gtest_filter which found no tests, so the exit code was zero,
indicating a passed test.
This bug has been here since r84387 in 2009, when Jeff Yasskin added the
original lit support for type-paratermized tests. Somewhere along the
line some of the ValueMapTests started failing, but we can fix those
separately.
llvm-svn: 234242
The internal shell was already threading around a 'cwd' parameter. We
just have to make it mutable so that we can update it as the test script
executes.
If the shell ever grows support for environment variable substitution,
we could also implement support for export.
llvm-svn: 231017
Summary: I think this is probably a bug, but I'm putting this up for review just to be sure. I think that `lit.util.capture` should decode the resulting string in the same way `lit.util.executeCommand` does.
Reviewers: ddunbar, EricWF
Reviewed By: EricWF
Subscribers: llvm-commits
Differential Revision: http://reviews.llvm.org/D6769
llvm-svn: 225681
Summary:
The following types can be encoded and decoded by the json library:
`dict`, `list`, `tuple`, `str`, `unicode`, `int`, `long`, `float`, `bool`, `NoneType`.
`JSONMetricValue` can be constructed with any of these types, and used as part of Test.Result.
This patch also adds a toMetricValue function that converts a value into a MetricValue.
Reviewers: ddunbar, EricWF
Reviewed By: EricWF
Subscribers: cfe-commits, llvm-commits
Differential Revision: http://reviews.llvm.org/D6576
llvm-svn: 224628
We were already requiring 2.5, which meant that people on old linux distros
had to upgrade anyway.
Requiring python 2.6 will make supporting 3.X easier as we can use the 3.X
exception syntax.
According to the discussion on llvmdev, there is not much value is requiring
just 2.6, we may as well just require 2.7.
llvm-svn: 224129
Summary:
This patch gives me just enough to leverage the existing functionality in `TestRunner` for use in `libc++` and `libc++abi` .
It does the following:
* Adds the `UNSUPPORTED` tag to `TestRunner.parseIntegratedTestScript`.
* Allows `parseIntegratedTestScript` to return an empty script if a script is not required by the caller.
Reviewers: ddunbar, EricWF
Reviewed By: EricWF
Subscribers: cfe-commits, llvm-commits
Differential Revision: http://reviews.llvm.org/D6589
llvm-svn: 223915
--xunit-xml-output saves test results to disk in JUnit's xml format. This will allow Jenkins to report the details of a lit run.
Based on a patch by David Chisnall.
llvm-svn: 223163
execution of a shell command. This can happen for example if the
``RUN:`` line calls a python script which can work correctly under
Linux/OSX but will not work under Windows. A more useful error message
is now shown rather than an unhelpful backtrace.
llvm-svn: 220227
This code is based on the existing LLVM Go bindings project hosted at:
https://github.com/go-llvm/llvm
Note that all contributors to the gollvm project have agreed to relicense
their changes under the LLVM license and submit them to the LLVM project.
Differential Revision: http://reviews.llvm.org/D5684
llvm-svn: 219976
As far as I can tell UTF-8 has been supported since the beginning of Python's
codec support, and it's the de facto standard for text these days, at least
for primarily-English text. This allows us to put Unicode into lit RUN lines.
rdar://problem/18311663
llvm-svn: 217688
Summary:
This patch changes the way xfail and unsupported tests are displayed.
This output is only displayed when the --show-unsupported/--show-xfail flags are passed to lit.
Currently xfail/unsupported tests are printed during the run of the test-suite. I think its better to display this information during the summary instead.
This patch removes the printing of these tests from when they are run to the summary.
Reviewers: ddunbar, EricWF
Reviewed By: EricWF
Subscribers: llvm-commits
Differential Revision: http://reviews.llvm.org/D4842
llvm-svn: 215809
Summary:
This patch add a --show-xfail flag. If this flag is specified then each xfail test will be printed to output.
When it is not given xfail tests are ignored. Ignoring xfail tests is the current behavior.
This flag is meant to mirror the --show-unsupported flag that was recently added.
Reviewers: ddunbar, EricWF
Reviewed By: EricWF
Subscribers: llvm-commits
Differential Revision: http://reviews.llvm.org/D4750
llvm-svn: 214609
It was pointed out that this breaks the "virtual test discovery"
mechanism, which allows for narming tests in the test exec root.
Reverting until I can figure out how to fix this.
llvm-svn: 211048
It would previously say things like
warning: input 'test/Frontend/foo.c' contained no tests
and have the user pull their hair trying to figure out what's wrong with that
file. This patch changes the message to the much clearer:
warning: no such file or directory: 'test/Frontend/foo.c'
Differential Revision: http://reviews.llvm.org/D4097
llvm-svn: 210597
Clang's lit cfg already detects the currently selected SDK via
"xcrun --show-sdk-path". The same thing should be done for compiler-rt tests,
to make them work on recent OS X versions. Instead of duplicating the detection
code, this patch extracts the detection function into a lit.util method.
Patch by Kuba Brecka (kuba.brecka@gmail.com),
reviewed at http://reviews.llvm.org/D4072
llvm-svn: 210534
This commit prospectively brings the benefits of r198766 to older supported
Python versions (2.5+).
Tested with Python 2.6, 2.7, 3.1 and 3.3 (!)
llvm-svn: 199009
On the other hand, exec(compile()) doesn't work in older Python versions in the
2.x series.
This commit introduces exec(compile()) with a fallback to plain exec(). That'll
hopefully hit the sweet spot in terms of version support.
Followup to r198766 which added enhanced source locations for lit cfg parsing.
llvm-svn: 199006
Python doesn't do a good job at diagnosing string exec() so use execfile()
where available.
This should be a timesaver when trying to get to the bottom of build bot
failures.
Before:
File "llvm/utils/lit/lit/TestingConfig.py", line 93, in load_from_path
exec("exec data in cfg_globals")
File "<string>", line 1, in <module>
File "<string>", line 194, in <module>
NameError: name 'typo' is not defined
After:
File "llvm/utils/lit/lit/TestingConfig.py", line 95, in load_from_path
execfile(path, cfg_globals)
File "clang/test/lit.cfg", line 194, in <module>
typo
^~~~
NameError: name 'typo' is not defined
llvm-svn: 198766
Add option -i to prioritize test runs by source file modification time and
previous failure state.
This optimal scheduling reduces typical test-and-fix iteration times to a
matter of seconds by rapidly answering the questions:
1) Did my recent change fix tests that were previously failing?
2) Do the tests I just wrote / modified still work?
The current implementation requires write permissions to the source tree
because it uses mtimes to track failures.
llvm-svn: 198150
The error raised by Python varies by platform(!), so let's just catch any
exception and fall back.
Thanks to Sylvestre Ledru for noticing this on a Debian / Python 2.7 system
running code coverage.
llvm-svn: 193516
so try PATH next. Assume it is sane enough to cover the usual system
bash locations too, but the old list is not good enough for NetBSD.
llvm-svn: 193471
If multiprocessing was requested, detected as available and subsequently failed
to initialize it's worth letting the user know about it before falling back to
threads.
This condition can arise in certain OpenBSD / FreeBSD Python versions.
llvm-svn: 193465
This should be a better fix for lit multiprocessing failures, replacing the
OpenBSD and FreeBSD workarounds in r193413 and r193457.
Reference: http://bugs.python.org/issue3770
llvm-svn: 193463
Speculative quick fix based on clang-X86_64-freebsd output:
File "/usr/local/lib/python2.6/multiprocessing/synchronize.py", line 33, in <module>
" function, see issue 3770.")
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.
llvm-svn: 193457
- This is a work-in-progress and all details are subject to change, but I am
trying to build up support for allowing lit to be used as a driver for
performance tests (or other tests which might want to record information
beyond simple PASS/FAIL).
llvm-svn: 190535
- This aligns with how existing test suites end up wanting to use the local
config files, conceptually it makes sense to consider them to be inherited.
llvm-svn: 189885
- At least on OS X, it is important for correct behavior of /bin/[ that argv[0]
is passed as written, and not as the full executable path.
llvm-svn: 189559
- For whatever reason, we have a lot of test files with bogus unicode
characters. This patch allows those scripts to still be parsed on Python3 by
changing the parsing logic to work on binary files, and only require the
actual script commands to be convertible to ascii.
- This patch has been tweaked to now ensure that the command strings are not of
unicode type on Python 2.6-7.
llvm-svn: 188398
- For whatever reason, we have a lot of test files with bogus unicode
characters. This patch allows those scripts to still be parsed on Python3 by
changing the parsing logic to work on binary files, and only require the
actual script commands to be convertible to ascii.
llvm-svn: 188376
- Injecting it as 'lit' is gross, since that name should be used to refer to
the actual package. For now both are available so it is possibly to cleanup
test config files incrementally.
llvm-svn: 188039
- Since we only have a few of these, use the cumbersome method of getting the
exception object from 'sys' to retain the current pre-2.6 compatibility.
llvm-svn: 187854
This change makes test with RUN lines like
RUN: opt ... | FileCheck
fail if opt fails, even if it prints what FileCheck wants. Enabling this
found some interesting cases of broken tests that were not being noticed
because opt (or some other tool) was crashing late.
Pipefail is used when the shell supports it or when using the internal
python based tester.
llvm-svn: 187261
The current machinery using KeyboardInterrupt for canceling doesn't work
with multiple threads on Windows as it just cancels the currently run tests
but the runners continue.
We install a handler for Ctrl-C which stops the provider from providing any
more tests to the runners. Together with aborting all currently running
tests, this brings lit to a halt.
llvm-svn: 186695
Native Windows Python will do line ending translation by default, which
we don't want in bash scripts. If we're not native Windows Python, then
'b' is ignored.
llvm-svn: 177602
RUN: a
RUN: b || true
as "a && (b || true)" in Tcl mode, and as "(a && b) || true" in sh mode.
Everyone seems to (quite reasonably) write tests assuming the Tcl behavior,
so use that in sh mode too.
llvm-svn: 169441
- The XTARGET feature (inherited from old DG tests) was just confusing (and
barely ever used). The same effect can now be achieved with a combination of
the more useful REQUIRES and XFAIL.
llvm-svn: 166305
'|&' bash syntax. We have lots of users with a bash on their system
which doesn't support this syntax, and as bash is still significantly
faster, we should support them.
The test suite has already been updated to cope with this.
llvm-svn: 159580
r145222 "lit/TestRunner.py: [Win32] Introduce WinWaitReleased(f), to wait for file handles to be released by children."
r145223 "lit/TestRunner.py: Use RemoveForce()."
r145381 "lit/TestRunner.py: Try to catch ERROR_FILE_NOT_FOUND, too."
r152916 "lit/TestRunner.py: [Win32] Check all opened_files[] released, rather than (obsoleted) written_files[]."
r153172 "lit/TestRunner.py: [Win32] Rework WinWaitReleased() again! "win32file" from Python Win32 Extensions."
llvm-svn: 156381
* Removed test/lib/llvm.exp - it is no longer needed
* Deleted the dg.exp reading code from test/lit.cfg. There are no dg.exp files
left in the test suite so this code is no longer required. test/lit.cfg is
now much shorter and clearer
* Removed a lot of duplicate code in lit.local.cfg files that need access to
the root configuration, by adding a "root" attribute to the TestingConfig
object. This attribute is dynamically computed to provide the same
information as was previously provided by the custom getRoot functions.
* Documented the config.root attribute in docs/CommandGuide/lit.pod
llvm-svn: 153408
This is in braces so that it doesn't conflict with the existing %p.
It uses braces instead of parens because parens would have to be
regex-escaped.
llvm-svn: 153213
We can simply confirm the handle released to open it with EXCLUSIVE. Attempting renaming was bad.
Disable win32file at ImportError. Thanks to Francois to let me know.
FIXME: Could we report warning or notification if win32file were not found?
llvm-svn: 153172
In previous case,
RUN: foo -o %t
RUN: FileCheck < %t
RUN: bar -o %t
2nd read handle might prevent manipulation of 3rd %t in bar, to remove and rename.
llvm-svn: 152916
1. Added a status note when a config file is loaded directly with load_config. This helps notice loads of lit.cfg from lit.site.cfg
2. Added a status note on the result of a config load. Previously, it was just notifying that it tries to load a config file. Now it will also say whether the load succeeded or the file wasn't found
The two changes give better visibility into which config files were actually loaded by lit. The effect is only on --debug runs.
Patch by Eli Bendersky!
llvm-svn: 149932
Kuba Mracek