This reverts commit bc3f54de18.
The patch breaks in the following two scenarios:
1. When manually passing an absolute path to llvm-lit with a lower-case
drive letter: `python bin\llvm-lit.py -sv c:\llvm-project\clang\test\PCH`
2. When the PWD has a lower-case drive letter, like after running
`cd c:\` with a lower-case "c:" (cmd's default is upper-case, but
it takes case-ness from what's passed to `cd` apparently).
There's been some back and forth if the cfg paths in the
config_map should be normcase()d. The argument for is that
it allows using all-lower spelling in cmd on Windows, the
argument against that doing so is lossy.
Before the relative-paths-in-generated-lit.site.cfg.py work,
there was no downside to calling normcase(), but with it
we need a hack to recover the original case.
This time, normcase() the hashtable key, but store the original
cased key in addition to the value. This fixes both cons, at the
cost of a few bytes more memory.
Differential Revision: https://reviews.llvm.org/D78169
Update local test object "in place" from remote test object. We need to
do this to ensure that discovered test object which is used for printing
test results reflect the changes.
> Why are we sending back the whole test object from the worker process
> (lit.worker.execute) instead of just the result?
Unfortunately, the test result is not the only "result" of test
execution. Other members (e.g., xfails, requires) of the Test class are
set only during execution. Those members affect the behavior of
`isExpectedToFail` and `setResult`, and are accessed when printing
results. For example, xunit.xml test results include missing features
for "skip reasons". The lack of separation between an immutable "test
definition" and "generated outputs" (including the primary result and
other secondary state) is unfortunate historical design decision in lit.
> Why do we update the initial test object instead of just discarding it
> and continuing with the pickled test object?
Both of these approaches would work. However, note that we need a fully
populated test object for printing results. Updating the existing one
seems to be the easier path.
We already print available features, and it can be useful to print
substitutions as well since those are a pretty fundamental part of
a test suite. We could also consider printing other things like the
test environment, however the need doesn't appear to be as strong.
As a fly-by fix, we also always print available features, even when
there are none.
Before:
$ lit -sv libcxx/test --show-suites
-- Test Suites --
libc++ - 6350 tests
Source Root: [...]
Exec Root : [...]
Available Features : -faligned-allocation -fsized-deallocation [...]
After:
$ lit -sv libcxx/test --show-suites
-- Test Suites --
libc++ - 6350 tests
Source Root: [...]
Exec Root : [...]
Available Features: -faligned-allocation -fsized-deallocation [...]
Available Substitutions: %{build_module} => [...]
%{build} => %{cxx} -o [...]
Differential Revision: https://reviews.llvm.org/D77818
max-time.py:
Windows does not have a native `sleep` command, use `time.sleep()` in
Python instead.
max-failures.py:
The max-failure test reused the shtest-shell test inputs instead of
defining its own "test domain". However, the output of this
shtest-shell "test domain" is slightly different on Windows, which now
bites us since we made the max-failures test stricter. Let's define
our own "max failures" test domain.
Fixup for cbe42a9d5f. Increase values for testing the overall lit
timeout (--max-time) which wasn't enough for the test to complete on
very slow build bots.
Track and print the number of skipped tests. Skipped tests are tests
that should have been executed but weren't due to:
* user interrupt [Ctrl+C]
* --max-time (overall lit timeout)
* --max-failures
This is part of a larger effort to ensure that all discovered tests are
properly accounted for.
Add test for overall lit timeout feature (`--max-time` option) to
observe skipped tests. Extend test for `--max-failures` option.
Reviewed By: jdenny
Differential Revision: https://reviews.llvm.org/D77819
lit supports `--time-tests` which will report the 20 slowest tests and
print a nice histogram for test times. This change prints this list and
the histogram rows by decreasing test times. After all, we are most
interested in the slowest tests.
The LitConfig is shared across the whole test suite. However, since
enabling recursive expansion can be a breaking change for some test
suites, it's important to confine the setting to test suites that
enable it explicitly.
Note that other issues were raised with the way recursiveExpansionLimit
operates. However, this commit simply moves the setting to the right
place -- the mechanism by which it works can be improved independently.
Differential Revision: https://reviews.llvm.org/D77415
Add a new overload of StaticLibraryDefinitionGenerator::Load that takes a triple
argument and supports loading archives from MachO universal binaries in addition
to regular archives.
The LLI tool is updated to use this overload.
Graceful lit shutdown on user keyboard interrupt [Ctrl+C] was a
longstanding goal of mine. After a few refactorings this revision
finally enables it. We use the following strategy to deal with
KeyboardInterrupt:
https://noswap.com/blog/python-multiprocessing-keyboardinterrupt
Printing of a helpful summary for interrupted runs (just as the one for
completed runs) will be tackled in future revisions.
Reviewed By: serge-sans-paille, rnk
Differential Revision: https://reviews.llvm.org/D77365
On shutdown, the result complete handler is not racing with the main
thread anymore because we are now always waiting for process pool
termination via
```
finally:
pool.join()
```
The shtest-inject test relied on being executed in "single process" mode
and started to fail with a `PicklingError` after it was removed:
```
Can't pickle <class 'lit.TestingConfig.CustomFormat'>: attribute
lookup lit.TestingConfig.CustomFormat failed
```
This happened because the test config has to be serialized to the worker
process, but apparently the `CustomFormat` class defined inline is not
serializable.
This change allows passing the tested functionality (preamble_commands)
directly to `lit.formats.ShTest` so we can use it directly in the test.
Remove the "serial run" abstraction which bypasses Python's
`multiprocessing.Pool` and instead directly runs tests without spawning
worker processes. This abstraction has not offered the benefits I hoped
it would and therefore does not carry its weight.
In previous commits [1,2] I changed worker.py to only send back the test
result from the worker process instead of the whole test object. This
was a mistake. lit.Test contains fields (e.g., xfials, requires,
unsupported) that are only populated when we actually execute the test,
but are queried when we report the results in the parent process. This
commit essentially reverts the following changes:
[1] a3d2f9b53a
[2] 17bb660fb8
A previous attempt to cleanup module imports broke installing via
pip/setup.py [1]. This should be fixed now.
[1] cf252240e8
Reviewed By: paquette
Differential Revision: https://reviews.llvm.org/D76940
This minor refactoring allows reducing the amount of processing that
is duplicated when we re-run a flaky test. It also has the nice
side effect that libc++'s current test format supports flaky .sh.cpp
tests, because those are built on top of _runShTest, not executeShTest.
This allows defining substitutions in terms of other substitutions. For
example, a %build substitution could be defined in terms of a %cxx
substitution as '%cxx %s -o %t.exe' and the script would be properly
expanded.
Differential Revision: https://reviews.llvm.org/D76178
This allows creating custom test formats on top of `executeShTest` that
inject commands at the beginning of the file being parsed, without
requiring these commands to physically appear in the test file itself.
For example, one could define a test format that prints out additional
debug information at the beginning of each test. More realistically,
this has been used to define custom test formats like one that supports
compilation failure tests (e.g. with the extension `compile.fail.cpp`)
by injecting a command that calls the compiler on the file itself and
expects it to fail.
Without this change, the only alternative is to create a temporary file
with the same content as the original test, then prepend the desired
`// RUN:` lines to that file, and call `executeShTest` on that file
instead. This is both slow and cumbersome to do.
Differential Revision: https://reviews.llvm.org/D76290
This commit adds a new keyword in lit called ALLOW_RETRIES. This keyword
takes a single integer as an argument, and it allows the test to fail that
number of times before it first succeeds.
This work attempts to make the existing test_retry_attempts more flexible
by allowing by-test customization, as well as eliminate libc++'s FLAKY_TEST
custom logic.
Differential Revision: https://reviews.llvm.org/D76288
Basically change the layout to please `go build` and remove references to
`llvm-go`.
Update llvm/test/Bindings/Go/ to use the system go compiler
Differential Revision: https://reviews.llvm.org/D74540
While looking at cycle time graphs of some of my bots, I noticed
that 327894859c made check-llvm noticeably slower on macOS and
Windows.
As it turns out, the 5 substitutions added in that change were
enough to cause lit to thrash the build-in cache in re.compile()
(re.sub() is implemented as re.compile().sub()), and apparently
applySubstitutions() is on the cricital path and slow when all
regexes need to compile all the time.
(See `_MAXCACHE = 512` in cpython/Lib/re.py)
Supporting full regexes for lit substitutions seems a bit like
overkill, but for now add a simple unbounded cache to recover
the lost performance.
No intended behavior change.
A refactoring commit (cf252240) removed this line. Removing it broke installing
lit with pip and setup.py.
This adds the line back in so that we can install lit again.
For an example of how this appeared, see:
http://green.lab.llvm.org/green/job/LNT_Tests/5853/
File "/Users/buildslave/jenkins/...s/__init__.py", line 2453, in resolve
raise ImportError(str(exc))
ImportError: 'module' object has no attribute 'main'
It would appear that the removal of this lit feature was incomplete
and there is a test case that still tests for this. This patch removes
the remaining tests to bring the bots back to green. I would encourage the
author to do a post-commit review on this in case there is a more desirable fix.
Summary:
Remove REQUIRES-ANY alias lit directive since it is hardly used and can
be easily implemented using an OR expression using REQUIRES. Fixup
remaining testcases still using REQUIRES-ANY.
Reviewers: probinson, jdenny, gparker42
Reviewed By: gparker42
Subscribers: eugenis, asb, rbar, johnrusso, simoncook, sabuasal, niosHD, delcypher, jrtc27, zzheng, edward-jones, rogfer01, MartinMosbeck, brucehoult, the_o, PkmX, jocewei, lenary, s.egerton, pzheng, sameer.abuasal, apazos, luismarques, cfe-commits, #sanitizers, llvm-commits
Tags: #llvm, #clang, #sanitizers
Differential Revision: https://reviews.llvm.org/D71408
E.g. the mingw python distributed in msys2 (the mingw one, which is a
normal win32 application and doesn't use the msys2 runtime itself),
despite being a normal win32 python, still uses forward slashes. This
works fine for other cases (many, but not all), but when constructing a
raw NT path, all path separators must be backslashes.
Differential Revision: https://reviews.llvm.org/D71490
Jenkins sometimes starts a new working directory by appending @2 (or
incrementing the number if the @n suffix is already there). This causes
several clang tests to fail as:
s@INPUT_DIR@%/S/Inputs@g
gets expanded to the invalid:
s@INPUT_DIR@/path/to/workdir@2/Inputs@g
~~~~~~~~~~
where the part marked with ~'s is interpreted as the flags. These are
invalid and the test fails.
Previous fixes simply exchanged the @ character for another like | but
that's just moving the problem. Address it by adding an expansion that
escapes the @ character we're using as a delimiter as well as other magic
characters in the replacement of sed's s@@@.
There's still room for expansions to cause trouble though. One I ran into
while testing this was that having a directory called foo@bar causes lots
of `CHECK-NOT: foo` directives to match. There's also things like
directories containing `\1`
Tests go through the following stages:
*) discovered
*) filtered
*) executed
Only executed tests have a result (e.g., PASS, FAIL, XFAIL, etc.). See
"result codes" in Test.py.
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D70612
Nico Weber