Latest upstream change in https://reviews.llvm.org/D117179 causes lit regressions on z/OS, when TMPDIR is exported and contains _, ld linker fails, it doesnt recognize _ specified in SYSLIN. this seems a limitation on z/OS. we need to fix lit.
Differential Revision: https://reviews.llvm.org/D118071
Upstream gtest now prints 'Running main() from FILE' instead of
plain 'Running main() from gtest_main.cc'. Thus, all such tests
ended-up being mistakenly marked as UNRESOLVED.
Patch by @lzaoral
Differential Revision: https://reviews.llvm.org/D100043
Currently the behavior with relative paths is pretty broken. It differs
between external shell and internal shell because the path resolution
is done with a different working directory. With the internal shell,
it's resolved relative to the directory from which lit is executed,
whereas with the external shell it's resolved relative to where the
test case is executed. To make matters worse, using the internal shell
the filepath to binaries looked up with `which` is returned relative
to the directory from which lit is executed, but then executed from
the test execution directory. That means that relative paths with the
internal shell give a `[Errno 2] No such file or directory` error
instead of the expected `command not found`.
To address these issues this patch makes lit interpret relative paths
as relative to the directory from which lit was invoked and modifies
`which` to return absolute paths, matching the behavior of its
namesake unix function.
See https://groups.google.com/g/llvm-dev/c/KzMWlOXR98Y/m/QJoqn0U5HAAJ
Reviewed By: yln
Differential Revision: https://reviews.llvm.org/D115486
When run in a git bash terminal, sys.stderr isn't flushed implicitly
after printing each line. Manually flush it after each printout,
to avoid getting broken/misordered output.
A similar fix had been done in the old libcxx test config, committed
as part of 7e3ee09ad2 / D28725; this
generalizes the fix, making it available in the new libcxx test
configs too, and for any other test that uses lit_config.note().
Differential Revision: https://reviews.llvm.org/D115761
This variable is necessary for the system() function for running
external processes.
This is needes for some libcxx tests. With the current libcxx test
infrastructure, all OS environment variables are passed through, but
with the new "from scratch" libcxx test setup, we only pass through
the variables listed here.
Differential Revision: https://reviews.llvm.org/D115363
LIT skips various system environment variables while building test
config. It turns out that we require PLATFORM environment variable for
detection of x86 vs Arm windows platform.
This patch adds system environment variable PLATFORM into LIT test
config for detection of win32 Arm platform.
Reviewed By: mstorsjo
Differential Revision: https://reviews.llvm.org/D113165
Add a new --order option to choose between available test orders:
the default "smart" order, predictable "lexical" order or "random"
order. Default to using lexical order and one job in the lit test
suite.
Differential Revision: https://reviews.llvm.org/D107695
This change adds the option --resultdb-output=path allow llvm-lit
generating LuCI ResultDB JSON output for the test results, which
can be better integrated with certain CI/CQ framework.
Differential Revision: https://reviews.llvm.org/D108238
AIX may use cat_64 for 64 bit cat, this is just update the lit test to accept the name as well.
Reviewed By: #powerpc, shchenz
Differential Revision: https://reviews.llvm.org/D108149
This reverts commit 3229c97151.
With a2acac6 in place this should provide enough info to work out
any repeat of the failure in cross_ovver_uniform_dist.test.
Currently the UNSUPPORTED and XFAIL clauses support specifying
substrings of the target triple; but REQUIRES does not, which can trip
people up or lead to hacking config files to insert substitute feature
names. Consistency across all three lit clauses seems preferable.
Differential Revision: https://reviews.llvm.org/D107162
Remove test times when running shtest-keyword-parse-errors test,
in order to prevent the previous executions from impacting subtest
order and therefore causing FileCheck to fail.
Differential Revision: https://reviews.llvm.org/D107427
For example, I need this lately in my CI config:
LIT_XFAIL_NOT='libomptarget :: nvptx64-nvidia-cuda :: unified_shared_memory/api.c'
That test specifies an XFAIL directive, but I get an XPASS result.
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D106022
The new documentation entry gives an example use case from
libomptarget.
Reviewed By: yln, jhenderson, davezarzycki
Differential Revision: https://reviews.llvm.org/D105208
This patch augments Lit with the ability to parse regular expressions
in boolean expressions. This includes REQUIRES:, XFAIL:, UNSUPPORTED:,
and all other special Lit markup that evaluates to a boolean expression.
Regular expressions can be specified by enclosing them in {{...}},
similarly to how FileCheck handles such regular expressions. The regular
expression can either be on its own, or it can be part of an identifier.
For example, a match expression like {{.+}}-apple-darwin{{.+}} would match
the following variables:
x86_64-apple-darwin20.0
arm64-apple-darwin20.0
arm64-apple-darwin22.0
etc...
In the long term, this could be used to remove the need to handle the
target triple specially when parsing boolean expressions.
Differential Revision: https://reviews.llvm.org/D104572
This is an attempt to fix clang test failures due to 'nonportable-include-path'
warnings on Windows when a path to llvm-project's base directory contains some
uppercase letters (excluding a drive letter).
The issue originates from 2 problems:
* discovery.py loads site config in lower case causing all the paths
based on __file__ and requested within the config file to be in lowercase as well,
* neither os.path.abspath() nor os.path.realpath() (both used to obtain paths of
config files, sources, object directories, etc) do not return paths in the correct
case for Windows (at least consistently for all python versions).
As os.path library doesn't seem to provide any relaible way to restore
the case for paths on Windows, this patch proposes to use pathlib.resolve().
pathlib is a part of Python 3.4 while llvm lit requires Python 3.6.
Reviewed By: Meinersbur
Differential Revision: https://reviews.llvm.org/D103014
In such cases, the executables are not in the llvm_tools_dir directory, so we need to look in the other search locations. Previously, they were found via the PATH, but this was disabled by default in commit rGa1e6565.
Depends on D103154.
Reviewed By: thopre
Differential Revision: https://reviews.llvm.org/D103156
The pipes.quote function quotes using single quotes, the same goes
for the newer shlex.quote (which is the preferred form in Python 3).
This isn't suitable for quoting in command lines on Windows (and the
documentation for shlex.quote even says it's only usable for Unix
shells).
In general, the python subprocess.list2cmdline function should do
proper quoting for the platform's current shell. However, it doesn't
quote the ';' char, which we pass within some arguments to run.py.
Therefore use the custom reimplementation from lit.TestRunner which
is amended to quote ';' too.
The fact that arguemnts were quoted with single quotes didn't matter
for command lines that were executed by either bash or the lit internal
shell, but if executing things directly using subprocess.call, as in
_supportsVerify, the quoted path to %{cxx} fails to be resolved by the
Windows shell.
This unlocks 114 tests that previously were skipped on Windows.
Differential Revision: https://reviews.llvm.org/D103310
This updates the googletest format to support tests that use GTEST_SKIP(),
which is now available with the updated googletest framework.
Differential Revision: https://reviews.llvm.org/D102694
This allows tests to detect whether to run or not, dependent on which
LLD version is required for the test.
Reviewed by: thopre
Differential Revision: https://reviews.llvm.org/D101997
This patch stops lit from looking on the PATH for clang, lld and other
users of use_llvm_tool (currently only the debuginfo-tests) unless the
call explicitly requests to opt into using the PATH. When not opting in,
tests will only look in the build directory.
See the mailing list thread starting from
https://lists.llvm.org/pipermail/llvm-dev/2021-May/150421.html.
See the review for details of why decisions were made about when still
to use the PATH.
Reviewed by: thopre
Differential Revision: https://reviews.llvm.org/D102630
Currently, if the user specifies the environment variable 'CLANG', tests
will attempt to use the value as a path to the clang executable.
Previously, lldb could also be specified via the CLANG environment
variable, but this was almost certainly a bug, because that meant both
clang and lldb would have the same path. This patch changes the
environment variable for lldb to 'LLDB'.
Reviewed by: thopre, teemperor
Differential Revision: https://reviews.llvm.org/D101982
At 61 or over, I see messages like
File "...\Python\Python39\lib\multiprocessing\connection.py", line 816, in _exhaustive_wait
res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 64
60 seems to work for me.
If this causes issues for anybody else, feel free to revert.
This reverts commit d319005a37.
Causing messages like:
File "...\Python\Python39\lib\multiprocessing\connection.py", line 816, in _exhaustive_wait
res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 74
Revert the 32-process cap on Windows. When testing with Swift, we found
that there was a time reduction for testing with the higher load. This
should hopefully not matter much in practice. In the case that the
original problem with python remains with a high subprocess count, we
can easily revert this change.
Previously, if the search_env argument was specified, and the tool was
found at that location, the path was not reported, unlike other
situations when this function was called. Adding the reporting makes the
function consistent.
Reviewed by: thopre
Differential Revision: https://reviews.llvm.org/D101896
A lit feature guards tests for the lit timeout functionality because on
most system it depends on the availability of the psutil Python module.
However, that feature is defined based on the ability of the testing lit
to cancel test, which does not necessarily apply to the ability of the
tested lit.
In particular, RUN commands have a cleared PYTHONPATH and user site
packages are disabled. In the case where psutil is found by the testing
lit from one of those two source of python path, the tested lit would
not be able to find it, causing timeout tests to fail.
This commit fixes the issue by testing the ability to cancel tests in
the RUN command environment.
Reviewed By: yln
Differential Revision: https://reviews.llvm.org/D99728
This fixes cases where "not not <command>" is supposed to return
only the error codes 0 or 1, but after efee57925c,
it passed the original error code through.
This was visible on AIX in the shtest-output-printing.py testcase,
where 'wc' returns 2, while it returns 1 on other platforms, and the
test required "not not" to normalize it to 1.
Keep running "not --crash" via the external "not" executable, but
for plain negations, and for cases that use the shell "!" operator,
just skip that argument and invert the return code.
The libcxx tests only use the shell operator "!" for negations,
never the "not" executable, because libcxx tests can be run without
having a fully built llvm tree available providing the "not"
executable.
This allows using the internal shell for libcxx tests.
It should be possible to reland this now that D99938 fixed the
one test failure in clang-tidy that broke when "not" was handled
internally, letting lit/python execute grep.exe directly instead
of via not.exe. (See D99330 and D99406 for more commentery on the
exact issue that broke and other potential ways of fixing it.)
Differential Revision: https://reviews.llvm.org/D98859
This avoids breaking clang-tidy/infrastructure/validate-check-names.cpp
if 'not' is evaluated as a lit internal tool (making TestRunner
invoke 'grep' directly in that test, instead of invoking 'not', which
then invokes 'grep').
The quoting of arguments is still brittle if the executable is an
MSYS based tool though, as MSYS based tools incorrectly unescape
backslashes in quoted arguments (contrary to regular win32 argument
parsing rules), see D99406 and
https://github.com/msys2/msys2-runtime/issues/36 for more examples
of the issues.
Differential Revision: https://reviews.llvm.org/D99938
Quality of progress bar and ETA in lit has always bothered me.
For example, given `./bin/llvm-lit /repositories/llvm-project/clang/test/CodeGen* -sv`
at 1%, it says it will take 10 more minutes,
at 25%, it says it will take 1.25 more minutes,
at 50%, it says it will take 30 more seconds,
and in the end finishes with `Testing Time: 39.49s`. That's rather wildly unprecise.
Currently, it assumes that every single test will take the same amount of time to run on average.
This is is a somewhat reasonable approximation overall, but it is quite clearly imprecise,
especially in the beginning.
But, we can do better now, after D98179! We now know how long the tests took to run last time.
So we can build a better ETA predictor, by accumulating the time spent already,
the time that will be spent on the tests for which we know the previous time,
and for the test for which we don't have previous time, again use the average time
over the tests for which we know current or previous run time.
It would be better to use median, but i'm wary of the cost that may incur.
Now, on **first** run of `./bin/llvm-lit /repositories/llvm-project/clang/test/CodeGen* -sv`
at 10%, it says it will take 30 seconds,
at 25%, it says it will take 50 more seconds,
at 50%, it says it will take 27 more seconds,
and in the end finishes with `Testing Time: 41.64s`. That's pretty reasonable.
And on second run of `./bin/llvm-lit /repositories/llvm-project/clang/test/CodeGen* -sv`
at 1%, it says it will take 1 minutes,
at 25%, it says it will take 30 more seconds,
at 50%, it says it will take 19 more seconds,
and in the end finishes with `Testing Time: 39.49s`. That's amazing i think!
I think people will love this :)
Reviewed By: yln
Differential Revision: https://reviews.llvm.org/D99073
Even though we have read the times before,
we intentionally forget about it for performance reasons.
But that means we also forget all the times for the tests
that weren't executed this time. This is mildly inconvenient.
So, when recording the new times, first re-read the old times,
and update times for the tests that were executed,
thus preserving all original times, too.
Jonas Devlieghere