Commit Graph

1024 Commits

Author SHA1 Message Date
Ilya Leoshkevich 0d89963df3 [Lit] Pass through DFLTCC from the surrounding environment
Many distros ship zlib with the IBM Z deflate hardware acceleration
patch [1]. Sometimes it's desirable to disable the acceleration, for
example, for reproducibility. This can be done by exporting DFLTCC=0.
llvm-lit clears this environment variable, which causes
compress-debug-sections-zlib.test fail on z15 and later machines. Add
DFLTCC to the list of variables to keep.

[1] https://github.com/madler/zlib/pull/410

Reviewed By: abrachet

Differential Revision: https://reviews.llvm.org/D130253
2022-07-23 01:38:30 +02:00
Yuanfang Chen e97b2d4138 [lit][unit] add a test for sanitizer-only test failures
Follow-up for 6b02c53936.
2022-07-15 18:36:38 -07:00
Yuanfang Chen ea7968bf76 [lit][test] explicitly use utf-8 encoding to write testing json file
Related test failure:
https://lab.llvm.org/buildbot/#/builders/178/builds/2527/steps/13/logs/FAIL__lit___googletest-timeout_py
2022-07-15 16:42:50 -07:00
Yuanfang Chen 6b02c53936 [lit][unit] make sure to include failed shard in test summary report
For sanitizer only failures, the tests all pass but still need to
put the shard in failed test summary.
2022-07-15 15:43:33 -07:00
Yuanfang Chen f32ccc2cca [lit] fix a bug in 4cd1c96d37
Only report failure for tests that actually runs.
2022-07-15 14:37:06 -07:00
Yuanfang Chen 4cd1c96d37 [reland][lit][unit] makes sure to print stderr when no test failures are found
In some situations, like running tests with sanitizers, all test
passes but the shard could still fail due to memory issues.

Relands 65769429c0
2022-07-15 13:26:30 -07:00
Yuanfang Chen 041bb94b9c Revert "[lit][unit] makes sure to print stderr when no test failures are found"
This reverts commit 65769429c0.

This patch is incomplete.
2022-07-15 13:26:30 -07:00
Yuanfang Chen 65769429c0 [lit][unit] makes sure to print stderr when no test failures are found
In some situations, like running tests with sanitizers, all test passes
but the shard could still fail due to memory issues.
2022-07-15 12:33:11 -07:00
Ben Langmuir 9044224575 [test] Fix macOS triple check
While the inferred host triple for macOS is something like
<arch>-apple-darwin, it's also valid to have <arch>-apple-macos.
Currently that globally changes whether an SDKROOT is provided in tests,
so make this check more portable.

Differential Revision: https://reviews.llvm.org/D129684
2022-07-13 13:36:16 -07:00
Yuanfang Chen 9d37895a71 [lit][test] relaxed GTEST_TOTAL_SHARDS checking for some googletests (2)
Missed this in 14d3021c10
2022-06-28 10:32:22 -07:00
Yuanfang Chen 14d3021c10 [lit][test] relaxed GTEST_TOTAL_SHARDS checking for some googletests
For machines with a small number of cores, GTEST_TOTAL_SHARDS may be
lower than 6.
2022-06-27 16:17:08 -07:00
Alex Brachet 98052b6463 Revert "[lit] Passthrough CLANG_MODULE_CACHE_PATH env var"
This reverts commit 6466c9abf3.
2022-06-12 03:01:08 +00:00
Alex Brachet 6466c9abf3 [lit] Passthrough CLANG_MODULE_CACHE_PATH env var
This environment variable can be set to control module
caching. It disables caching by setting the variable
empty. As such, it needs to be handled differently
from other environment variables here which are
assumed to not be empty.
2022-06-11 21:04:02 +00:00
Vitaly Buka d33c36235d [lit] Fix setup of sanitizer environment
Not all options were propageted into tests.

Reviewed By: ychen

Differential Revision: https://reviews.llvm.org/D122869
2022-05-19 19:24:16 -07:00
Sam McCall 1236b66a98 [lit] pass LLVM_SYMBOLIZER_PATH through to tests.
Currently several buildbots give unsymbolized traces on crash.
I suspect these are configuring the symbolizer in this way and regressed in
D122251 or thereabouts.

Trying this coupled with a reland of patch that failed on a couple of bots with
no useful stacktrace...
2022-05-18 15:30:37 +02:00
David Goldman e91a73de24 [Lit] Add pushd and popd builtins
This behaves just like the sh/cmd.exe equivalents.

pushd/popd are useful to verify path handling of the driver,
typically testing prefix maps or relative path handling.

Differential Revision: https://reviews.llvm.org/D125502
2022-05-12 19:24:30 -04:00
Yuanfang Chen 0e554ebf02 [lit][unit] set the default result start and pid
In case of interrupting, there were None. Fixes PR55176.
2022-04-28 13:39:11 -07:00
Andrew Savonichev 1041a9642b [lit] Support %if ... %else syntax for RUN lines
This syntax allows to modify RUN lines based on features
available. For example:

    RUN: ... | FileCheck %s --check-prefix=%if windows %{CHECK-W%} %else %{CHECK-NON-W%}
    CHECK-W: ...
    CHECK-NON-W: ...

The whole command can be put under %if ... %else:

    RUN: %if tool_available %{ %tool %} %else %{ true %}

or:

    RUN: %if tool_available %{ %tool %}

If tool_available feature is missing, we'll have an empty command in
this RUN line.  LIT used to emit an error for empty commands, but now
it treats such commands as nop in all cases.

Multi-line expressions are also supported:

    RUN: %if tool_available %{ \
    RUN:   %tool               \
    RUN: %} %else %{           \
    RUN:   true                \
    RUN: %}

Background and motivation:
D121727 [NVPTX] Integrate ptxas to LIT tests
https://reviews.llvm.org/D121727

Differential Revision: https://reviews.llvm.org/D122569
2022-04-27 20:29:08 +03:00
Yuanfang Chen d3efa577f5 [lit] Keep stdout/stderr when using GoogleTest format
When a unit test crashes or timeout, print the shard's stdout and
stderr. When a unit test fails, attaches the test's output to the LIT
output to help debugging.

While at it, concatenating shard's environment variables using space
instead of newline to make the reproducer script user friendly.

Based on D123797. (Thanks to @lenary)
2022-04-25 12:26:03 -07:00
Vitaly Buka b61f173f24 [lit] Forward more sanitizer env in TestingConfig 2022-04-15 00:50:52 -07:00
Vitaly Buka 43e0cedc1e [NFC] Reformat a part of TestingConfig.py 2022-04-15 00:50:52 -07:00
Yuanfang Chen cd0a5889d7 [Reland][lit] Use sharding for GoogleTest format
This helps lit unit test performance by a lot, especially on windows. The performance gain comes from launching one gtest executable for many subtests instead of one (this is the current situation).

The shards are executed by the test runner and the results are stored in the
json format supported by the GoogleTest. Later in the test reporting stage,
all test results in the json file are retrieved to continue the test results
summary etc.

On my Win10 desktop, before this patch: `check-clang-unit`: 177s, `check-llvm-unit`: 38s; after this patch: `check-clang-unit`: 37s, `check-llvm-unit`: 11s.
On my Linux machine, before this patch: `check-clang-unit`: 46s, `check-llvm-unit`: 8s; after this patch: `check-clang-unit`: 7s, `check-llvm-unit`: 4s.

Reviewed By: yln, rnk, abrachet

Differential Revision: https://reviews.llvm.org/D122251
2022-04-12 14:51:12 -07:00
Alex Brachet 47f59df892 Revert "Reland "[lit] Use sharding for GoogleTest format""
This reverts commit 948f3deca9.
2022-04-04 16:34:28 +00:00
Yuanfang Chen 948f3deca9 Reland "[lit] Use sharding for GoogleTest format"
This relands commit a87ba5c86d.

Adjust llvm/utils/lit/tests/googletest-timeout.py for new test output.
2022-04-03 22:35:45 -07:00
Yuanfang Chen c0f90c84b1 Revert "[lit] Use sharding for GoogleTest format"
This reverts commit a87ba5c86d.

Breaks bots:
https://lab.llvm.org/buildbot/#/builders/196/builds/10454
2022-04-03 20:04:55 -07:00
Yuanfang Chen a87ba5c86d [lit] Use sharding for GoogleTest format
This helps lit unit test performance by a lot, especially on windows. The performance gain comes from launching one gtest executable for many subtests instead of one (this is the current situation).

The shards are executed by the test runner and the results are stored in the
json format supported by the GoogleTest. Later in the test reporting stage,
all test results in the json file are retrieved to continue the test results
summary etc.

On my Win10 desktop, before this patch: `check-clang-unit`: 177s, `check-llvm-unit`: 38s; after this patch: `check-clang-unit`: 37s, `check-llvm-unit`: 11s.
On my Linux machine, before this patch: `check-clang-unit`: 46s, `check-llvm-unit`: 8s; after this patch: `check-clang-unit`: 7s, `check-llvm-unit`: 4s.

Reviewed By: yln, rnk

Differential Revision: https://reviews.llvm.org/D122251
2022-04-03 19:47:02 -07:00
Sam McCall 75acad41bc Use lit_config.substitute instead of foo % lit_config.params everywhere
This mechanically applies the same changes from D121427 everywhere.

Differential Revision: https://reviews.llvm.org/D121746
2022-03-16 09:57:41 +01:00
Sam McCall e4cab4e24d [lit] Remove clang_libs_dir
Seems to have been added accidentally in 58db03a116 and then
copied into clangd by me (but not actually needed).
2022-03-16 03:03:02 +01:00
Sam McCall fa1019fa18 [lit] add lit_config.substitute to interpolate lit_config.params
A version of this logic appears in ~every lit.site.cfg.in (28 copies total).
This patch just removes two, but I'll update the rest of llvm-project next.

Differential Revision: https://reviews.llvm.org/D121427
2022-03-15 22:09:43 +01:00
Keith Smiley 6541d3e979 [test] Add lit helper for windows paths
This adds 2 new lit helpers `%{fs-src-root}` and `%{fs-sep}`, these
allow writing tests that correctly handle slashes on Windows. In the
case of tests like clang/test/CodeGen/debug-prefix-map.c, these are
unable to correctly test behavior on both platforms, unless they fork
and add OS requirements, because the relevant logic hits host specific
codepaths like checking if paths are absolute.

Differential Revision: https://reviews.llvm.org/D111457
2022-03-14 20:05:55 -07:00
Sam McCall 895653d904 [cmake] Remove unused OUTPUT_MAPPING param to configure_lit_site_cfg(). NFC
Looks like it never did anything.
Added in 71deeee593
2022-03-10 21:26:53 +01:00
Martin Storsjö 9e2236fb7b Reapply [lit] Read command stdout/stderr as text on Windows
This takes care of normalizing newlines back to single LF instead
of CRLF.

Fix up a couple tests that accidentally pass binary data to stdout.

Differential Revision: https://reviews.llvm.org/D120623
2022-03-03 13:31:31 +02:00
Martin Storsjö f570da28e6 Revert "[lit] Read command stdout/stderr as text on Windows"
This reverts commit 640e45b9b2.

That commit seemed to quite conclusively break a number of buildbots.
2022-03-01 14:43:55 +02:00
Martin Storsjö 640e45b9b2 [lit] Read command stdout/stderr as text on Windows
This takes care of normalizing newlines back to single LF instead
of CRLF.

This on itself breaks on a couple tests that accidentally seem to
be writing binary data to stdout; make sure those cases are piped
to /dev/null instead of actually written to a terminal.

Differential Revision: https://reviews.llvm.org/D120623
2022-03-01 14:24:02 +02:00
Douglas Yung 98504fbb3d Make shtest-format.py CHECK lines more flexible
The test sometimes fails on Windows due to a warning emitted by bash about not
being able to find the /tmp directory causing this test to randomly fail. This
update makes the test more flexible to account for this possibility and should
hopefully make it more reliable.

Reviewed By: probinson

Differential Revision: https://reviews.llvm.org/D118691
2022-02-10 13:50:10 -08:00
Jonas Devlieghere 43d3d886a0 [llvm] Recognize arm64 as target-aarch64 in lit
Recognize arm64 triples as AArch64 so we can XFAIL/skip tests using
target-aarch64.

Differential revision: https://reviews.llvm.org/D119169
2022-02-08 21:29:21 -08:00
Tom Stellard a2601c9887 Bump the trunk major version to 15 2022-02-01 23:54:52 -08:00
Nancy Wang 22b0fe3fd9 [SystemZ][z/OS]: fix lit tmp_dir to use - instead of _
Latest upstream change in https://reviews.llvm.org/D117179 causes lit regressions on z/OS, when TMPDIR is exported and contains _, ld linker fails, it doesnt recognize _ specified in SYSLIN. this seems a limitation on z/OS. we need to fix lit.

Differential Revision: https://reviews.llvm.org/D118071
2022-01-24 17:47:48 -05:00
Lukáš Zaoral 14a793ab05 [lit] Fix compatibility with upstream gtest
Upstream gtest now prints 'Running main() from FILE' instead of
plain 'Running main() from gtest_main.cc'. Thus, all such tests
ended-up being mistakenly marked as UNRESOLVED.

Patch by @lzaoral

Differential Revision: https://reviews.llvm.org/D100043
2022-01-19 08:02:28 -08:00
Julian Lettner 0f6f6284d7 [lit] Make sure our test temp directory is actually used by tests
All credit to Martin Storsjö (mstorsjo) who describes the issue here:
https://github.com/llvm/llvm-project/issues/53167

Differential Revision: https://reviews.llvm.org/D117179
2022-01-13 11:23:32 -08:00
Geoffrey Martin-Noble e8b5b72182 [lit] Support relative path arguments
Currently the behavior with relative paths is pretty broken. It differs
between external shell and internal shell because the path resolution
is done with a different working directory. With the internal shell,
it's resolved relative to the directory from which lit is executed,
whereas with the external shell it's resolved relative to where the
test case is executed. To make matters worse, using the internal shell
the filepath to binaries looked up with `which` is returned relative
to the directory from which lit is executed, but then executed from
the test execution directory. That means that relative paths with the
internal shell give a `[Errno 2] No such file or directory` error
instead of the expected `command not found`.

To address these issues this patch makes lit interpret relative paths
as relative to the directory from which lit was invoked and modifies
`which` to return absolute paths, matching the behavior of its
namesake unix function.

See https://groups.google.com/g/llvm-dev/c/KzMWlOXR98Y/m/QJoqn0U5HAAJ

Reviewed By: yln

Differential Revision: https://reviews.llvm.org/D115486
2021-12-20 11:49:42 -08:00
Martin Storsjö 2ec75a0869 [lit] Flush stderr manually on Windows after printing messages
When run in a git bash terminal, sys.stderr isn't flushed implicitly
after printing each line. Manually flush it after each printout,
to avoid getting broken/misordered output.

A similar fix had been done in the old libcxx test config, committed
as part of 7e3ee09ad2 / D28725; this
generalizes the fix, making it available in the new libcxx test
configs too, and for any other test that uses lit_config.note().

Differential Revision: https://reviews.llvm.org/D115761
2021-12-18 21:41:40 +02:00
Martin Storsjö aff028f7d8 [lit] Pass the COMSPEC variable through to test processes on Windows
This variable is necessary for the system() function for running
external processes.

This is needes for some libcxx tests. With the current libcxx test
infrastructure, all OS environment variables are passed through, but
with the new "from scratch" libcxx test setup, we only pass through
the variables listed here.

Differential Revision: https://reviews.llvm.org/D115363
2021-12-08 23:29:39 +02:00
Muhammad Omair Javaid cbd215dfe5 [LIT] Add win32 PLATFORM env var to test config
LIT skips various system environment variables while building test
config. It turns out that we require PLATFORM environment variable for
detection of x86 vs Arm windows platform.

This patch adds system environment variable PLATFORM into LIT test
config for detection of win32 Arm platform.

Reviewed By: mstorsjo

Differential Revision: https://reviews.llvm.org/D113165
2021-11-04 17:42:43 +05:00
Michał Górny 2f69c82cec [llvm] [lit] Support forcing lexical test order
Add a new --order option to choose between available test orders:
the default "smart" order, predictable "lexical" order or "random"
order.  Default to using lexical order and one job in the lit test
suite.

Differential Revision: https://reviews.llvm.org/D107695
2021-08-27 20:47:11 +02:00
serge-sans-paille 46c947af7e Have lit preserve SOURCE_DATE_EPOCH
This environment variable has been standardized for reproducible builds. Setting
it can help to have reproducible tests too, so keep it as part of the testing
env when set.

See https://reproducible-builds.org/docs/source-date-epoch/

Differential Revision: https://reviews.llvm.org/D108332
2021-08-25 10:38:20 +02:00
Haowei Wu 253cb50c60 [lit] Add the option to output test result as resultdb json format
This change adds the option --resultdb-output=path allow llvm-lit
generating LuCI ResultDB JSON output for the test results, which
can be better integrated with certain CI/CQ framework.

Differential Revision: https://reviews.llvm.org/D108238
2021-08-23 17:00:50 -07:00
Jinsong Ji 3c4fad6a69 [LIT]Accept cat_64 command name on AIX in shtest
AIX may use cat_64 for 64 bit cat, this is just update the lit test to accept the name as well.

Reviewed By: #powerpc, shchenz

Differential Revision: https://reviews.llvm.org/D108149
2021-08-17 14:05:30 +00:00
Paul Robinson 5b2037fa95 Revert "Reapply "[lit] Have REQUIRES support the target triple""
This reverts commit 187c69e9ef.
compiler-rt/test/cross_over_uniform_dist.test refuses to pass.
2021-08-09 12:14:57 -07:00
Paul Robinson 187c69e9ef Reapply "[lit] Have REQUIRES support the target triple"
This reverts commit 3229c97151.

With a2acac6 in place this should provide enough info to work out
any repeat of the failure in cross_ovver_uniform_dist.test.
2021-08-09 06:43:52 -07:00