This change adds the option --resultdb-output=path allow llvm-lit
generating LuCI ResultDB JSON output for the test results, which
can be better integrated with certain CI/CQ framework.
Differential Revision: https://reviews.llvm.org/D108238
AIX may use cat_64 for 64 bit cat, this is just update the lit test to accept the name as well.
Reviewed By: #powerpc, shchenz
Differential Revision: https://reviews.llvm.org/D108149
This reverts commit 3229c97151.
With a2acac6 in place this should provide enough info to work out
any repeat of the failure in cross_ovver_uniform_dist.test.
Currently the UNSUPPORTED and XFAIL clauses support specifying
substrings of the target triple; but REQUIRES does not, which can trip
people up or lead to hacking config files to insert substitute feature
names. Consistency across all three lit clauses seems preferable.
Differential Revision: https://reviews.llvm.org/D107162
Remove test times when running shtest-keyword-parse-errors test,
in order to prevent the previous executions from impacting subtest
order and therefore causing FileCheck to fail.
Differential Revision: https://reviews.llvm.org/D107427
For example, I need this lately in my CI config:
LIT_XFAIL_NOT='libomptarget :: nvptx64-nvidia-cuda :: unified_shared_memory/api.c'
That test specifies an XFAIL directive, but I get an XPASS result.
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D106022
The new documentation entry gives an example use case from
libomptarget.
Reviewed By: yln, jhenderson, davezarzycki
Differential Revision: https://reviews.llvm.org/D105208
This patch augments Lit with the ability to parse regular expressions
in boolean expressions. This includes REQUIRES:, XFAIL:, UNSUPPORTED:,
and all other special Lit markup that evaluates to a boolean expression.
Regular expressions can be specified by enclosing them in {{...}},
similarly to how FileCheck handles such regular expressions. The regular
expression can either be on its own, or it can be part of an identifier.
For example, a match expression like {{.+}}-apple-darwin{{.+}} would match
the following variables:
x86_64-apple-darwin20.0
arm64-apple-darwin20.0
arm64-apple-darwin22.0
etc...
In the long term, this could be used to remove the need to handle the
target triple specially when parsing boolean expressions.
Differential Revision: https://reviews.llvm.org/D104572
This is an attempt to fix clang test failures due to 'nonportable-include-path'
warnings on Windows when a path to llvm-project's base directory contains some
uppercase letters (excluding a drive letter).
The issue originates from 2 problems:
* discovery.py loads site config in lower case causing all the paths
based on __file__ and requested within the config file to be in lowercase as well,
* neither os.path.abspath() nor os.path.realpath() (both used to obtain paths of
config files, sources, object directories, etc) do not return paths in the correct
case for Windows (at least consistently for all python versions).
As os.path library doesn't seem to provide any relaible way to restore
the case for paths on Windows, this patch proposes to use pathlib.resolve().
pathlib is a part of Python 3.4 while llvm lit requires Python 3.6.
Reviewed By: Meinersbur
Differential Revision: https://reviews.llvm.org/D103014
In such cases, the executables are not in the llvm_tools_dir directory, so we need to look in the other search locations. Previously, they were found via the PATH, but this was disabled by default in commit rGa1e6565.
Depends on D103154.
Reviewed By: thopre
Differential Revision: https://reviews.llvm.org/D103156
The pipes.quote function quotes using single quotes, the same goes
for the newer shlex.quote (which is the preferred form in Python 3).
This isn't suitable for quoting in command lines on Windows (and the
documentation for shlex.quote even says it's only usable for Unix
shells).
In general, the python subprocess.list2cmdline function should do
proper quoting for the platform's current shell. However, it doesn't
quote the ';' char, which we pass within some arguments to run.py.
Therefore use the custom reimplementation from lit.TestRunner which
is amended to quote ';' too.
The fact that arguemnts were quoted with single quotes didn't matter
for command lines that were executed by either bash or the lit internal
shell, but if executing things directly using subprocess.call, as in
_supportsVerify, the quoted path to %{cxx} fails to be resolved by the
Windows shell.
This unlocks 114 tests that previously were skipped on Windows.
Differential Revision: https://reviews.llvm.org/D103310
This updates the googletest format to support tests that use GTEST_SKIP(),
which is now available with the updated googletest framework.
Differential Revision: https://reviews.llvm.org/D102694
This allows tests to detect whether to run or not, dependent on which
LLD version is required for the test.
Reviewed by: thopre
Differential Revision: https://reviews.llvm.org/D101997
This patch stops lit from looking on the PATH for clang, lld and other
users of use_llvm_tool (currently only the debuginfo-tests) unless the
call explicitly requests to opt into using the PATH. When not opting in,
tests will only look in the build directory.
See the mailing list thread starting from
https://lists.llvm.org/pipermail/llvm-dev/2021-May/150421.html.
See the review for details of why decisions were made about when still
to use the PATH.
Reviewed by: thopre
Differential Revision: https://reviews.llvm.org/D102630
Currently, if the user specifies the environment variable 'CLANG', tests
will attempt to use the value as a path to the clang executable.
Previously, lldb could also be specified via the CLANG environment
variable, but this was almost certainly a bug, because that meant both
clang and lldb would have the same path. This patch changes the
environment variable for lldb to 'LLDB'.
Reviewed by: thopre, teemperor
Differential Revision: https://reviews.llvm.org/D101982
At 61 or over, I see messages like
File "...\Python\Python39\lib\multiprocessing\connection.py", line 816, in _exhaustive_wait
res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 64
60 seems to work for me.
If this causes issues for anybody else, feel free to revert.
This reverts commit d319005a37.
Causing messages like:
File "...\Python\Python39\lib\multiprocessing\connection.py", line 816, in _exhaustive_wait
res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 74
Revert the 32-process cap on Windows. When testing with Swift, we found
that there was a time reduction for testing with the higher load. This
should hopefully not matter much in practice. In the case that the
original problem with python remains with a high subprocess count, we
can easily revert this change.
Previously, if the search_env argument was specified, and the tool was
found at that location, the path was not reported, unlike other
situations when this function was called. Adding the reporting makes the
function consistent.
Reviewed by: thopre
Differential Revision: https://reviews.llvm.org/D101896
A lit feature guards tests for the lit timeout functionality because on
most system it depends on the availability of the psutil Python module.
However, that feature is defined based on the ability of the testing lit
to cancel test, which does not necessarily apply to the ability of the
tested lit.
In particular, RUN commands have a cleared PYTHONPATH and user site
packages are disabled. In the case where psutil is found by the testing
lit from one of those two source of python path, the tested lit would
not be able to find it, causing timeout tests to fail.
This commit fixes the issue by testing the ability to cancel tests in
the RUN command environment.
Reviewed By: yln
Differential Revision: https://reviews.llvm.org/D99728
This fixes cases where "not not <command>" is supposed to return
only the error codes 0 or 1, but after efee57925c,
it passed the original error code through.
This was visible on AIX in the shtest-output-printing.py testcase,
where 'wc' returns 2, while it returns 1 on other platforms, and the
test required "not not" to normalize it to 1.
Keep running "not --crash" via the external "not" executable, but
for plain negations, and for cases that use the shell "!" operator,
just skip that argument and invert the return code.
The libcxx tests only use the shell operator "!" for negations,
never the "not" executable, because libcxx tests can be run without
having a fully built llvm tree available providing the "not"
executable.
This allows using the internal shell for libcxx tests.
It should be possible to reland this now that D99938 fixed the
one test failure in clang-tidy that broke when "not" was handled
internally, letting lit/python execute grep.exe directly instead
of via not.exe. (See D99330 and D99406 for more commentery on the
exact issue that broke and other potential ways of fixing it.)
Differential Revision: https://reviews.llvm.org/D98859
This avoids breaking clang-tidy/infrastructure/validate-check-names.cpp
if 'not' is evaluated as a lit internal tool (making TestRunner
invoke 'grep' directly in that test, instead of invoking 'not', which
then invokes 'grep').
The quoting of arguments is still brittle if the executable is an
MSYS based tool though, as MSYS based tools incorrectly unescape
backslashes in quoted arguments (contrary to regular win32 argument
parsing rules), see D99406 and
https://github.com/msys2/msys2-runtime/issues/36 for more examples
of the issues.
Differential Revision: https://reviews.llvm.org/D99938
Quality of progress bar and ETA in lit has always bothered me.
For example, given `./bin/llvm-lit /repositories/llvm-project/clang/test/CodeGen* -sv`
at 1%, it says it will take 10 more minutes,
at 25%, it says it will take 1.25 more minutes,
at 50%, it says it will take 30 more seconds,
and in the end finishes with `Testing Time: 39.49s`. That's rather wildly unprecise.
Currently, it assumes that every single test will take the same amount of time to run on average.
This is is a somewhat reasonable approximation overall, but it is quite clearly imprecise,
especially in the beginning.
But, we can do better now, after D98179! We now know how long the tests took to run last time.
So we can build a better ETA predictor, by accumulating the time spent already,
the time that will be spent on the tests for which we know the previous time,
and for the test for which we don't have previous time, again use the average time
over the tests for which we know current or previous run time.
It would be better to use median, but i'm wary of the cost that may incur.
Now, on **first** run of `./bin/llvm-lit /repositories/llvm-project/clang/test/CodeGen* -sv`
at 10%, it says it will take 30 seconds,
at 25%, it says it will take 50 more seconds,
at 50%, it says it will take 27 more seconds,
and in the end finishes with `Testing Time: 41.64s`. That's pretty reasonable.
And on second run of `./bin/llvm-lit /repositories/llvm-project/clang/test/CodeGen* -sv`
at 1%, it says it will take 1 minutes,
at 25%, it says it will take 30 more seconds,
at 50%, it says it will take 19 more seconds,
and in the end finishes with `Testing Time: 39.49s`. That's amazing i think!
I think people will love this :)
Reviewed By: yln
Differential Revision: https://reviews.llvm.org/D99073
Even though we have read the times before,
we intentionally forget about it for performance reasons.
But that means we also forget all the times for the tests
that weren't executed this time. This is mildly inconvenient.
So, when recording the new times, first re-read the old times,
and update times for the tests that were executed,
thus preserving all original times, too.
I.e. when you first run lit on a directory, and then on a single test,
the timing knowledge about anything else other than that single test
is lost. This isn't right.
All of these depend on the order of tests, so if one runs them twice,
the tests within them will naturally be reordered
using the previous run times, which breaks them.
If lit was run on a directory that contained no suites,
then naturally suite[0] will not be there,
and that line would cause python warnings.
So just predicate it with a check that it is there in the first place.
This reverts commit d09adfd399.
That commit caused failures in
clang-tidy/infrastructure/validate-check-names.cpp on windows
buildbots.
That change exposed a surprising issue, not directly related to
this change in itself, but in how TestRunner quotes command line
arguments that later are going to be interpreted by a msys based
tool (like grep.exe, when provided by Git for Windows). This
worked accidentally before, when grep was invoked via not.exe
which took a more conservative approach to windows argument quoting.
When running in a Windows Container, the Git for Windows Unix tools
(C:\Program Files\Git\usr\bin) just hang if this variable isn't
passed through.
Currently, running the LLVM/clang tests in a Windows Container fails
if that directory is added to the path, but succeeds after this change.
(After this change, the previously used GnuWin tools can be left out
entirely, too, as lit automatically picks up the Git for Windows tools
if necessary.)
Differential Revision: https://reviews.llvm.org/D98858
Keep running "not --crash" via the external "not" executable, but
for plain negations, and for cases that use the shell "!" operator,
just skip that argument and invert the return code.
The libcxx tests only use the shell operator "!" for negations,
never the "not" executable, because libcxx tests can be run without
having a fully built llvm tree available providing the "not"
executable.
This allows using the internal shell for libcxx tests.
Differential Revision: https://reviews.llvm.org/D98859
The "path" recorded for timing purposes is only used as a key into a dictionary. It is never used as an actual path to a filesystem API, therefore we should use '/' as the canonical separator so that Unix and Windows machines can share timing data. This also ensures that the lit testing works across platforms.
Reviewed By: jhenderson, jmorse
Differential Revision: https://reviews.llvm.org/D98767
The test file has embedded slashes. This is fine for normal users that
are just recording and reordering paths, but not great when the trace
data is committed back to a repository that should work on both Unix and
Windows.
Lit as it exists today has three hacks that allow users to run tests earlier:
1) An entire test suite can set the `is_early` boolean.
2) A very recently introduced "early_tests" feature.
3) The `--incremental` flag forces failing tests to run first.
All of these approaches have problems.
1) The `is_early` feature was until very recently undocumented. Nevertheless it still lacks testing and is a imprecise way of optimizing test starting times.
2) The `early_tests` feature requires manual updates and doesn't scale.
3) `--incremental` is undocumented, untested, and it requires modifying the *source* file system by "touching" the file. This "touch" based approach is arguably a hack because it confuses editors (because it looks like the test was modified behind the back of the editor) and "touching" the test source file doesn't work if the test suite is read only from the perspective of `lit` (via advanced filesystem/build tricks).
This patch attempts to simplify and address all of the above problems.
This patch formalizes, documents, tests, and defaults lit to recording the execution time of tests and then reordering all tests during the next execution. By reordering the tests, high core count machines run faster, sometimes significantly so.
This patch also always runs failing tests first, which is a positive user experience win for those that didn't know about the hidden `--incremental` flag.
Finally, if users want, they can _optionally_ commit the test timing data (or a subset thereof) back to the repository to accelerate bots and first-time runs of the test suite.
Reviewed By: jhenderson, yln
Differential Revision: https://reviews.llvm.org/D98179
Visual Studios implementation of the C++ Standard Library does not use strerror to produce a message for std::error_code unlike other standard libraries such as libstdc++ or libc++ that might be used.
This patch adds a cmake script that through running a C++ program gets the error messages for the POSIX error codes and passes them onto lit through an optional config parameter.
If the config parameter is not set, or getting the messages failed, due to say a cross compiling configuration without an emulator, it will fall back to using pythons strerror functions.
Differential Revision: https://reviews.llvm.org/D98278
This patch uses the errno python library to print out the correct error messages instead of hardcoding the error message per platform.
Reviewed By: jhenderson, ASDenysPetrov
Differential Revision: https://reviews.llvm.org/D97472
For some build configurations, `check-all` calls lit multiple times to
run multiple lit test suites. Most recently, I've found this to be
true when configuring openmp as part of `LLVM_ENABLE_RUNTIMES`, but
this is not the first time.
If one test suite fails, none of the remaining test suites run, so you
cannot determine if your patch has broken them. It can then be
frustrating to try to determine which `check-` targets will run the
remaining tests without getting stuck on the failing tests.
When such cases arise, it is probably best to adjust the cmake
configuration for `check-all` to run all test suites as part of one
lit invocation. Because that fix will likely not be implemented and
land immediately, this patch introduces `--ignore-fail` to serve as a
workaround for developers trying to see test results until it does
land:
```
$ LIT_OPTS=--ignore-fail ninja check-all
```
One problem with `--ignore-fail` is that it makes it challenging to
detect test failures in a script, perhaps in CI. This problem should
serve as motivation to actually fix the cmake configuration instead of
continuing to use `--ignore-fail` indefinitely.
Reviewed By: jhenderson, thopre
Differential Revision: https://reviews.llvm.org/D96371
In semi-automated environments, XFAILing or filtering out known regressions without actually committing changes or temporarily modifying the test suite can be quite useful.
Reviewed By: yln
Differential Revision: https://reviews.llvm.org/D96662
With enough cores, the slowest tests can significantly change the total testing time if they happen to run late. With this change, a test suite can improve performance (for high-end systems) by listing just a few of the slowest tests up front.
Reviewed By: jdenny, jhenderson
Differential Revision: https://reviews.llvm.org/D96594
Some test systems do not use lit for test discovery but only for its
substitution and test selection because they use another way of managing
test collections, e.g. CTest. This forces those tests to be invoked with
lit --no-indirectly-run-check. When a mix of lit version is in use, it
requires to detect the availability of that option.
This commit provides a new config option standalone_tests to signal a
directory made of tests meant to run as standalone. When this option is
set, lit skips test discovery and the indirectly run check. It also adds
the missing documentation for --no-indirectly-run-check.
Reviewed By: jdenny
Differential Revision: https://reviews.llvm.org/D94766
On z/OS, other error messages are not matched correctly in lit tests.
```
EDC5121I Invalid argument.
EDC5111I Permission denied.
```
This patch adds a lit substitution to fix it.
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D95808
On z/OS, the following error message is not matched correctly in lit tests.
```
EDC5129I No such file or directory.
```
This patch uses a lit config substitution to check for platform specific error messages.
Reviewed By: muiez, jhenderson
Differential Revision: https://reviews.llvm.org/D95246
The initial problem with the remaining bot config was resolved.
We can now use Python3. Let's use `os.cpu_count()` to cleanup this
helper.
Differential Revision: https://reviews.llvm.org/D94734
Update shebang to always use Python3 when executing `lit.py` directly.
A previous change of mine [1] revealed that we still use Python2 on some
bot configurations that invoke `llvm/utils/lit/lit.py` as a script
directly (instead of `python3 path/to/lit.py`).
[1] https://reviews.llvm.org/D94734
Differential Revision: https://reviews.llvm.org/D95393
In addition to consistency, we'll hit a wall when 11.1.0 gets released, because
we cannot represent it with lit versioning scheme.
Differential Revision: https://reviews.llvm.org/D94157
Historically, we have told contributors that GnuWin32 is a pre-requisite
because our tests depend on utilities such as sed, grep, diff, and more.
However, Git on Windows includes versions of these utilities in its
installation. Furthermore, GnuWin32 has not been updated in many years.
For these reasons, it makes sense to have the ability to run llvm tests
in a way that is both:
a) Easier on the user (less stuff to install)
b) More up-to-date (The verions that ship with git are at least as
new, if not newer, than the versions in GnuWin32.
We add support for this here by attempting to detect where Git is
installed using the Windows registry, confirming the existence of
several common Unix tools, and then adding this location to lit's PATH
environment.
Differential Revision: https://reviews.llvm.org/D84380
Don't produce or expect any output from the infinite looping test -
doing so is a recipe for racey flakyness without a longer timeout to
ensure the output is received first, even though that doesn't seem
integral/important to the test. Instead have a plain, no output infinite
loop and check that that is caught and handled.
If for some reason the output is valuable for test coverage - the
timeout should be increased from 1 second to give the process time to
output the text, flush, and for that text to be received and buffered
before the test is timed out.
On some of the slow or heavily-loaded bots, this test was failing
intermittently because the infinite_loop.py script might not emit
anything to stdout before the 1 second timeout, so the "Command Output"
line isn't present in the output. That output isn't really important to
this test, we just care that the process is killed, so we can just rmove
that check line from the test.
Differential revision: https://reviews.llvm.org/D92563
- pass required=False to use_clang(), as we don't need it
- fix required=False (which was unused and rotted):
- make derived substitutions conditional on it
- add a feature so we can disable tests that need it
- conditionally disable our one test that depends on %resource_dir.
This doesn't seem right from first principles, but isn't a big deal.
Differential Revision: https://reviews.llvm.org/D90528
Currently, a LIT test named directly (on the command line) will
be run even if the name of the test file does not meet the rules
to be considered a test in the LIT test configuration files for
its test suite. For example, if the test does not have a
recognised file extension.
This makes it relatively easy to write a LIT test that won't
actually be run. I did in: https://reviews.llvm.org/D82567
This patch adds an error to avoid users doing that. There is a
small performance overhead for this check. A command line option
has been added so that users can opt into the old behaviour.
Differential Revision: https://reviews.llvm.org/D83069
I did some profiling of lit while trying to optimize the libc++ test
startup for remote hosts and it turns out that there is a realpath() call
for every message printed and this shows up in the profile.
The inspect.getframeinfo() function calls realpath() internally and
moreover we don't need most of the other information returned from it.
This patch uses inspect.getsourcefile() and os.path.abspath() to remove
../ from the path instead. Not resolving symlinks reduces the startup time
for running a single test with lit by about 50ms for me.
Reviewed By: ldionne, yln
Differential Revision: https://reviews.llvm.org/D89186
I recently had to run the test suite with a debug Python which got started
warning about some invalid escape sequences in LLDB's Python code. They all
attempt to add a backslash by doing a single backslash instead of a double
backslash in a normal string. This seems to work fine for now, but Python says
this behaviour is deprecated, so this patch turns all those strings into raw
strings (where a single backslash is actually a single backslash)
Reviewed By: JDevlieghere
Differential Revision: https://reviews.llvm.org/D88289
The tests previously relied on the `short.py` and `FirstTest.subTestA`
script being executed on a machine within a short time window (1 or 2
seconds). While this "seems to work" it can fail on resource constrained
machines. We could bump the timeout a little bit (bumping it too
much would mean the test would take a long time to execute) but it wouldn't
really solve the problem of the test being prone to failures.
This patch tries to remove this flakeyness by separating testing into
two separate parts:
1. Testing if a test can hit a timeout.
2. Testing if a test can run to completion in the presence of a
timeout.
This way we can give (1.) a really short timeout (to make the test run
as fast as possible) and (2.) a really long timeout. This means for (2.)
we are no longer trying to rely on the "short" test executing within
some short time window. Instead the window is now 3600 seconds which
should be long enough even for a heavily resource constrained machine to
execute the "short" test.
Thanks to Julian Lettner for suggesting this approach. This superseeds
my original approach in https://reviews.llvm.org/D88807.
This patch also changes the command line override test to run the quick
test rather than the slow one to make the test run faster.
Differential Revision: https://reviews.llvm.org/D89020
This reverts b3418cb4eb and d12ae042e1
This breaks some external bots, see discussion in https://reviews.llvm.org/D84380
In the meanwhile, please use `cmake -DLLVM_LIT_TOOLS_DIR="C:/Program Files/Git/usr/bin"` or add it to %PATH%.
Summary:
This patch adds an error to Clang that detects if OpenMP offloading is
used between two architectures with incompatible pointer sizes. This
ensures that the data mapping can be done correctly and solves an issue
in code generation generating the wrong size pointer. This patch adds a
new lit substitution, %omp_powerpc_triple that, if the system is 32-bit or
64-bit, sets the powerpc triple accordingly. This was required to fix
some OpenMP tests that automatically populated the target architecture.
Reviewers: jdoerfert
Subscribers: cfe-commits guansong sstefan1 yaxunl delcypher
Tags: OpenMP clang LLVM
Differential Revision: https://reviews.llvm.org/D88594
This should fix the test failures on the clang win64 bot:
http://lab.llvm.org:8011/builders/clang-x64-windows-msvc/builds/18830
It has been red since Sept 23-ish.
This was subtle to debug. Windows has 'find' and 'sort' utilities in
C:\Windows\system32, but they don't support all the same flags as the
coreutils programs. I configured the buildbot above with Python 2.7
64-bit (hey, it was set up in 2016). When I installed git for Windows, I
opted to add all the Unix utilities that come with git to the system
PATH. This is *almost* enough to make the LLVM tests pass, but not
quite, because if you use the system PATH, the Windows version of find
and sort come first, but the tests that use diff, cmp, etc, will all
pass. So only a handful of tests will fail, and with cryptic error
messages.
The code changed in this CL doesn't work with Python 2. Before
Python 3.2, the winreg.OpenKey function did not accept the `access=`
keyword argument, the caller was required to pass an unused `reserved`
positional argument of 0. The try/except/pass around the OpenKey
operation masked this usage error in Python 2.
Further, the result of the registry operation has to be converted from
unicode to add it to the environment, but that was incidental.
Historically, we have told contributors that GnuWin32 is a pre-requisite
because our tests depend on utilities such as sed, grep, diff, and more.
However, Git on Windows includes versions of these utilities in its
installation. Furthermore, GnuWin32 has not been updated in many years.
For these reasons, it makes sense to have the ability to run llvm tests
in a way that is both:
a) Easier on the user (less stuff to install)
b) More up-to-date (The verions that ship with git are at least as
new, if not newer, than the versions in GnuWin32.
We add support for this here by attempting to detect where Git is
installed using the Windows registry, confirming the existence of
several common Unix tools, and then adding this location to lit's PATH
environment.
Differential Revision: https://reviews.llvm.org/D84380
`clang` currently requires the native linker on Solaris:
- It passes `-C` to `ld` which GNU `ld` doesn't understand.
- To use `gld`, one needs to pass the correct `-m EMU` option to select
the right emulation. Solaris `ld` cannot handle that option.
So far I've worked around this by passing `-DCLANG_DEFAULT_LINKER=/usr/bin/ld`
to `cmake`. However, if someone forgets this, it depends on the user's
`PATH` whether or not `clang` finds the correct linker, which doesn't make
for a good user experience.
While it would be nice to detect the linker flavor at runtime, this is more
involved. Instead, this patch defaults to `/usr/bin/ld` on Solaris. This
doesn't work on its own, however: a link fails with
clang-12: error: unable to execute command: Executable "x86_64-pc-solaris2.11-/usr/bin/ld" doesn't exist!
I avoid this by leaving absolute paths alone in `ToolChain::GetLinkerPath`.
Tested on `amd64-pc-solaris2.11`, `sparcv9-sun-solaris2.11`, and
`x86_64-pc-linux-gnu`.
Differential Revision: https://reviews.llvm.org/D84029
Failing test output sometimes contains control characters like \x1b (e.g.
if there was some -fcolor-diagnostics output) which are not allowed inside
XML files. This causes problems with CI systems: for example, the Jenkins
JUnit XML will throw an exception when ecountering those characters and
similar problems also occur with GitLab CI.
Reviewed By: yln, jdenny
Differential Revision: https://reviews.llvm.org/D84233
This produces a chrome://tracing compatible trace file in the same way
as -ftime-trace.
This can be useful in optimising test time where one long test is causing
long overall test time on a wide machine.
This also helped in finding tests which have side effects on others
(e.g. https://reviews.llvm.org/D84885).
Differential Revision: https://reviews.llvm.org/D84931
Otherwise, if a Lit script contains escaped substitutions (like %%p in this test https://github.com/llvm/llvm-project/blob/master/compiler-rt/test/asan/TestCases/Darwin/asan-symbolize-partial-report-with-module-map.cpp#L10), they are unescaped during recursive application of substitutions, and the results are unexpected.
We solve it using the fact that double percent signs are first replaced with #_MARKER_#, and only after all the other substitutions have been applied, #_MARKER_# is replaced with a single percent sign. The only change is that instead of replacing #_MARKER_# at each recursion step, we replace it once after the last recursion step.
Differential Revision: https://reviews.llvm.org/D83894
When running multiple shards, don't include skipped tests in the xunit
output since merging the files will result in duplicates.
In our CHERI Jenkins CI, I configured the libc++ tests to run using sharding
(since we are testing using a single-CPU QEMU). We then merge the generated
XUnit xml files to produce a final result, but if the individual XMLs
report tests excluded due to sharding each test is included N times in the
final result. This also makes it difficult to find the tests that were
skipped due to missing REQUIRES: etc.
Reviewed By: yln
Differential Revision: https://reviews.llvm.org/D84235
Summary:
As a corrollary, these tests are now run as part of the check-flang
target.
Reviewers: sscalpone
Subscribers: mgorny, delcypher, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D83946
Summary: In Python3 SubstituteCaptures are no longer converted to String implicitly behind the scenes. Converting explicitly makes the TestRunner to work in Python3.
Reviewers: gribozavr2, compnerd
Reviewed By: gribozavr2
Subscribers: tbkka, delcypher, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D81361
As per discussion in D69207, have lit ignore UnicodeDecodeErrors
when running with python 2 in an ASCII shell.
Differential Revision: https://reviews.llvm.org/D82754
Provide `--show-xxx` flags for all non-failure result codes, just as we
already do for `--show-xfail` and `--show-unsupported`.
Reviewed By: jdenny
Differential Revision: https://reviews.llvm.org/D82233
Haowei Wu