When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.
When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.
To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.
Reviewed By: asmith, delcypher
Differential Revision: https://reviews.llvm.org/D44598
llvm-svn: 330755
`shtest-xunit-output.py` test.
Although there is no `-` file Jeremy Morse has reported to me that it
causes problems in their setup because lit tries to find it and ends up
loading an out of tree lit configuration file.
llvm-svn: 330728
The test is apparently needed e.g. for check-cfi on Windows where we get
'C:/b/slave/sanitizer-windows/build/./bin/clang.exe': command not found
without it. Try to fix the problem that was fixed by r330672 by also checking
for isabs() instead.
llvm-svn: 330673
lit's util.which() would check if the passed-in path existed directly,
and if so return it as-is. This is never the case when running llvm's, clang's,
or lld's tests normally. But when running `./llvm-lit path/to/clang/test`
with a cwd of llvm-build/bin, this if would detect that clang exists at path
'clang' and return 'clang' as the discovered clang binary -- and then lit would
use the " clang " -> "*** Do not use 'clang' in tests, use '%clang'. ***"
substitution to replace that with a broken test. By removing this early
return, lit ends up with the usual absolute path and everything works even
in this uncommon case.
llvm-svn: 330672
XML printer.
A test has been added that tries to comprehensively test emitting
XUnit XML output for shell tests.
Differential Revision: https://reviews.llvm.org/D45567
llvm-svn: 330409
There are two versions of to_string used by TestRunner.py. The one defined
in TestRunner.py and the one defined in utils/lit/lit/util.py. The util.py
version is superior to the TestRunner.py version.
This change removes the duplicate to_string in TestRunner.py in favor of
always using the version from util.py. Beside removing duplicate code, this
makes it easier to debug TestRunner.py since only one version of to_string
is used.
Patch by Stella Stamenova!
llvm-svn: 329972
- In Python 2.x, basestring is the base string type, but in
Python 3.x basestring is not defined and instead str includes
unicode strings.
- When Python is in a path that includes spaces, it needs to
be specified with quotes in the test files for it to run.
- The cache.ll test relies on files of a specific size being
created by Python, but on some versions of Windows the
files that are created by the current code are one byte
larger than expected. To fix the test, update file creation
to always make files of the expected size.
Patch by Stella Stamenova!
llvm-svn: 329466
LLVM Bug Id : 36449
Revision 328563 caused tests to fail under python 3.
This patch modified cat.py file to support both python 2 and 3.
This patch also fixes CRLF issues on Windows.
Patch by Chamal de Silva
Differential Revision: https://reviews.llvm.org/D45077
llvm-svn: 329123
Only rely on Python 3 (io.open) when necessary. This puts TestRunnyer.py closer to how it behaved
before the changes introduced in D43165 and silences a few Windows build bot failures.
Thanks to Stella Stamenova for the patch!
llvm-svn: 329037
Reapply D43165 which was reverted because of different versions of python failing.
The one line fix for the different python versions was commited at the same time
that D43165 was reverted. If this change is giving you issues then get in touch
with your python version and we will fix it.
llvm-svn: 329022
Summary:
This issue was found when running the clang unit test on Windows. Python 3.x cannot open some of the files that the tests are using with a simple open because of their encoding. Python 2.7+ and Python 3.x both support io.open which allows for an encoding to be specified.
This change will determine whether two files being compared should be opened (and then compared) as text or binary and whether to use utf-8 or the default encoding before proceeding with a line-by-line comparison.
Patch by Stella Stamenova!
Reviewers: zturner, llvm-commits, rnk, MaggieYi
Reviewed By: zturner
Subscribers: asmith, MatzeB, stella.stamenova, delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D43165
llvm-svn: 329012
This reverts commit 771829b640a5494ab65c810dd6b4330522bf3a33 (rr328598)
Hopefully the test will now pass on the bots.
rdar://problem/38774530
llvm-svn: 328703
The `shtest-timeout.py` test was failing intermittently. It looks like
the issue is that on a resource constrained system lit is unable to run
`quick_then_slow.py` twice and print out the messages the tests expects
within the one second timeout.
The underlying issue is that the test is dependent on the performance of
the host machine is a rather fragile way. This is due to hardcoding
timeout values and having assumptions that the host machine is able to
perform a certain amount of work within the hardcoded timeout values.
We could increase the timeout values but that doesn't really fix the
underlying issue. Instead this patch removes one of fragile assumptions
in the hope that this will be enough to fix the bots.
There are other fragile assumptions in this test (e.g. `quick.py` can be
executed in less than 1 second). If the bots continue to fail we'll have
to revisit this.
rdar://problem/38774530
llvm-svn: 328702
Summary:
This reverts commit r328596.
Checking if the arguments are strings before testing if they contain "/dev/null".
Reviewers: rnk
Reviewed By: rnk
Subscribers: delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D44914
llvm-svn: 328603
Summary:
These changes are to allow to a Result object to have nested Result objects in
order to support microbenchmarks. Currently lit is restricted to reporting one
result object for one test, this change provides support tests that want to
report individual timings for individual kernels.
This revision is the result of the discussions in
https://reviews.llvm.org/D32272#794759,
https://reviews.llvm.org/D37421#f8003b27 and https://reviews.llvm.org/D38496.
It is a separation of the changes purposed in https://reviews.llvm.org/D40077.
This change will enable adding LCALS (Livermore Compiler Analysis Loop Suite)
collection of loop kernels to the llvm test suite using the google benchmark
library (https://reviews.llvm.org/D43319) with tracking of individual kernel
timings.
Previously microbenchmarks had been handled by using macros to section groups
of microbenchmarks together and build many executables while still getting a
grouped timing (MultiSource/TSVC). Recently the google benchmark library was
added to the test suite and utilized with a litsupport plugin. However the
limitation of 1 test 1 result limited its use to passing a runtime option to
run only 1 microbenchmark with several hand written tests
(MicroBenchmarks/XRay). This runs the same executable many times with different
hand-written tests. I will update the litsupport plugin to utilize the new
functionality (https://reviews.llvm.org/D43316).
These changes allow lit to report micro test results if desired in order to get
many precise timing results from 1 run of 1 test executable.
Reviewers: MatzeB, hfinkel, rengolin, delcypher
Differential Revision: https://reviews.llvm.org/D43314
llvm-svn: 327422
On Windows, if the substitution contains a back reference, it would
removed due to the replacement of the escape character in lit. Create a
helper class to avoid this which will simply ignore the replacement and
mark the substitution as having capture groups being referenced.
llvm-svn: 327082
We are running lld tests with "--full-shutdown" option because we don't
want to call _exit() in lld if it is running tests. Regular shutdown
is needed for leak sanitizer.
This patch changes the way how we tell lld that it is running tests.
Now "--full-shutdown" is removed, and LLD_IN_TEST environment variable
is used instead.
This patch enables full shutdown on all ports, e.g. ELF, COFF and wasm.
Previously, we enabled it only for ELF.
Differential Revision: https://reviews.llvm.org/D43410
llvm-svn: 325413
Allow CLANG environment variable be copied into the testing configuration
and proper support testing with a custom path to the clang executable.
Signed-off-by: Vladimir Vereschaka <vvereschaka@accesssoftek.com>
llvm-svn: 324706
Summary:
That would allow to recursively compare directories in tests using
"diff -r" on Windows in a similar way as it can be done on Linux or Mac.
Reviewers: zturner, morehouse, vsk
Reviewed By: zturner
Subscribers: kcc, llvm-commits
Differential Revision: https://reviews.llvm.org/D41776
llvm-svn: 322102
The variable named `minor` was actually pointing to the patch part of
the version. While I was changing this I also made the check for Apple
clang more robust by checking both patch and minor rather than just
minor.
llvm-svn: 319656
The latest clang that ships with Xcode (clang 900 or 9.0.0) does not
support LSan. This fixes the lit configuration to reflect that.
Differential revision: https://reviews.llvm.org/D40672
llvm-svn: 319530
This is still breaking greendragon.
At this point I give up until someone can fix the greendragon
bots, and I will probably abandon this effort in favor of using
a private github repository.
llvm-svn: 318722
This was reverted due to the tests being run twice on some
build bots. Each run had a slightly different configuration
due to the way in which it was being invoked. This fixes
the problem (albeit in a somewhat hacky way). Hopefully in
the future we can get rid of the workflow of running
debuginfo-tests as part of clang, and then this hack can
go away.
llvm-svn: 318697
Summary:
Currently, LIT configures the LLVM binary path before the Clang binary path. However this breaks testing out-of-tree Clang builds (where the LLVM binary path includes a copy of Clang).
This patch reverses the order of the paths when looking for Clang, putting the Clang binary directory first.
Reviewers: zturner, beanz, chapuni, modocache, EricWF
Reviewed By: EricWF
Subscribers: mgorny, cfe-commits, llvm-commits
Differential Revision: https://reviews.llvm.org/D40217
llvm-svn: 318607
This is still broken because it causes certain tests to be
run twice with slightly different configurations, which is
wrong in some cases.
You can observe this by running:
ninja -nv check-all | grep debuginfo-tests
And seeing that it passes clang/test and clang/test/debuginfo-tests
to lit, which causes it to run debuginfo-tests twice. The fix is
going to involve either:
a) figuring out that we're running in this "deprecated" configuration,
and then deleting the clang/test/debuginfo-tests path, which should
cause it to behave identically to before, or:
b) make lit smart enough that it doesn't descend into a sub-suite if
that sub-suite already has a lit.cfg file.
llvm-svn: 318486
This was reverted due to some failures on specific darwin buildbots,
the issue being that the new lit configuration was not setting the
SDKROOT environment variable. We've tested a fix locally and confirmed
that it works, so this patch resubmits everything with the fix
applied.
llvm-svn: 318435
This reverts the aforementioned patch and 2 subsequent follow-ups,
as some buildbots are still failing 2 tests because of it.
Investigation is ongoing into the cause of the failures.
llvm-svn: 318112
Previously, debuginfo-tests was expected to be checked out into
clang/test and then the tests would automatically run as part of
check-clang. This is not a standard workflow for handling
external projects, and it brings with it some serious drawbacks
such as the inability to depend on things other than clang, which
we will need going forward.
The goal of this patch is to migrate towards a more standard
workflow. To ease the transition for build bot maintainers,
this patch tries not to break the existing workflow, but instead
simply deprecate it to give maintainers a chance to update
the build infrastructure.
Differential Revision: https://reviews.llvm.org/D39605
llvm-svn: 317925
I need a test that only runs in a reasonable amount of time on systems
that have sparse files. The broadest class of systems that support
sparse files are linux systems. So restricting my test to linux systems
should suffice. This change adds the system-linux feature to llvm-lit so
that it can be required.
Differential Revision: https://reviews.llvm.org/D39482
llvm-svn: 317055
The new scheme should match the normalization of embedded paths in
linkrepro tar files.
Differential Revision: https://reviews.llvm.org/D39023
llvm-svn: 316044
Reid Kleckner