Commit Graph

131 Commits

Author SHA1 Message Date
Stella Stamenova cc2404c01d [lit, python] Always add quotes around the python path in lit
Summary:
The issue with the python path is that the path to python on Windows can contain spaces. To make the tests always work, the path to python needs to be surrounded by quotes.

This change updates several configuration files which specify the path to python as a substitution and also remove quotes from existing tests.

Reviewers: asmith, zturner, alexshap, jakehehrlich

Reviewed By: zturner, alexshap, jakehehrlich

Subscribers: mehdi_amini, nemanjai, eraman, kbarton, jakehehrlich, steven_wu, dexonsmith, stella.stamenova, delcypher, llvm-commits

Differential Revision: https://reviews.llvm.org/D50206

llvm-svn: 339073
2018-08-06 22:37:44 +00:00
Justin Bogner 78bb0b4b3c [lit] Move the shtest-xunit-output check lines into shtest-format
These two tests are operating on the same test suite, which causes
them to be racy about writing temporary files and can cause spurious
failures. Merge them into one test to avoid the issue.

llvm-svn: 337718
2018-07-23 18:08:36 +00:00
Hans Wennborg c4b7e0125f Relax shtest-run-at-line.py
The test was failing on Windows machines which had bash.exe on PATH (but
not in the so called lit tools dir, containing cmp.exe, grep.exe etc.).

The problem was that the outer lit invocation would load LLVMConfig
from utils/lit/lit/llvm/config.py, which looks up the tools path with
getToolsPath(). That has a surprising side effect of also setting
bashPath, in our case setting it to empty.

The outer lit invocation would thus configure the pdbg0 and pdbg1
substitutions based on not running with bash.

But the inner lit invocation would not load LLVMConfig, so bash
would be found on PATH, that would be used as external shell,
and so the output wouldn't match pdbg0 and pdbg1.

It seems weird to me that getBashPath() will return different results
depending on whether getToolsPath() has been called before, but I
also don't know how to fix it properly.

This commit just relaxes the test case, because there doesn't seem
to be much point in testing for the exact syntax of the run file
as long as it works.

(See https://crbug.com/850023)

llvm-svn: 334100
2018-06-06 14:53:03 +00:00
Joel E. Denny 44ee15f34f [lit] Fix windows cmd.exe test config for r333620
llvm-svn: 333630
2018-05-31 05:48:33 +00:00
Joel E. Denny fc01dd281d [lit] Terminate ": RUN at line N" with ";" not "&&"
This fixes projects/compiler-rt/test/fuzzer/sigusr.test, which was
broken by r333614.  The trouble was that "&&" changes the command for
which "$!" gives the pid.

llvm-svn: 333620
2018-05-31 03:40:37 +00:00
Joel E. Denny 31b373963f [lit] Report line number for failed RUN command
(Relands r333584, reverted in 333592.)

When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed.  For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.

When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.

To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.

To support reporting RUN line numbers in the case of windows cmd.exe
as the external shell, this patch extends -vv to set "echo on" instead
of "echo off" in bat files.  (Support for windows cmd.exe as a lit
external shell will likely be dropped later, but I found out too
late.)

Reviewed By: delcypher,	asmith, stella.stamenova, jmorse, lebedev.ri, rnk

Differential Revision: https://reviews.llvm.org/D44598

llvm-svn: 333614
2018-05-31 00:55:32 +00:00
Joel E. Denny 71792c741e Revert r333584: [lit] Report line number for failed RUN command
It breaks test-suite.

llvm-svn: 333592
2018-05-30 21:07:27 +00:00
Joel E. Denny b6423479a1 [lit] Report line number for failed RUN command
(Relands r330755 (reverted in r330848) with fix for PR37239.)

When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed.  For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.

When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.

To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.

To support reporting RUN line numbers in the case of windows cmd.exe
as the external shell, this patch extends -vv to set "echo on" instead
of "echo off" in bat files.  (Support for windows cmd.exe as a lit
external shell will likely be dropped later, but I found out too
late.)

Reviewed By: delcypher,	asmith, stella.stamenova, jmorse, lebedev.ri, rnk

Differential Revision: https://reviews.llvm.org/D44598

llvm-svn: 333584
2018-05-30 19:42:27 +00:00
Chris Matthews 29770f7a62 Use quoteattr to ensure we make well formed attributes
We were making malformed XML on tests with ' in the name.  Switch to
using saxutils to set all of our attributes, so it can handle quotes
etc correctly.

llvm-svn: 333249
2018-05-25 00:31:36 +00:00
Dan Liew 6dfcc78364 [lit] Try to make `shtest-timeout.py` test more reliable by using a
larger timeout value. This really isn't very good because it will
still be susceptible to machine performance.

While we are here also fix a bug in validation of
`maxIndividualTestTime` where previously it wasn't checked if the
type was an int.

rdar://problem/40221572

llvm-svn: 332987
2018-05-22 15:06:29 +00:00
Dan Liew 9f7786855c [lit] Don't run `slow.py` in `shtest-timeout.py` test.
The program used to be used in `quick_then_slow.py` but that was
removed in r328702. The tests always run `slow.py` on its own but
this doesn't really test additional code so we'll just drop running
`slow.py` so the tests run faster.

rdar://problem/40221572

llvm-svn: 332986
2018-05-22 15:06:24 +00:00
Dan Liew 8b4d36a530 [lit] Don't check output of commands used in `shtest-timeout.py` test.
If the system is under heavy load 1 second might not be long enough
for it to produce output which could lead to spurious test failures.
What matters is that the right test cases reach a timeout.

rdar://problem/40221572

llvm-svn: 332985
2018-05-22 15:06:20 +00:00
Douglas Yung 2dd62a3d99 Mark test with "REQUIRES: shell" since it directly invokes "sh" and was failing on Windows.
llvm-svn: 332563
2018-05-17 01:36:25 +00:00
Alexander Richardson 1bffd0f741 Escape ]]> in xunit xml output
Summary:
This sequence ends the CDATA block so any characters after that are no
longer escaped. This can be fixed by replacing "]]>" with "]]]]><![CDATA[>".

Reviewers: cmatthews

Reviewed By: cmatthews

Differential Revision: https://reviews.llvm.org/D46886

llvm-svn: 332440
2018-05-16 09:00:28 +00:00
Chris Matthews 48357c43d5 remove output xml incase it is leftover from another run
llvm-svn: 332424
2018-05-16 00:37:00 +00:00
Chris Matthews 5d1014bff5 Use not to catch unexpected pass as well as remove old test results
As per review feedback, make sure we rm temp files, and make the return
code checking for lit more specific.

llvm-svn: 332423
2018-05-16 00:33:29 +00:00
Chris Matthews 811a83d036 Add the message attribute to skipped
JUnit xml allows for a message attribute to be displayed on skips. Lets
populate that with an analysis of why we skipped the test.

llvm-svn: 332156
2018-05-11 23:15:11 +00:00
Chris Matthews d1386a88dd [LIT] replace output escapes wit a cdata block
CDATA blocks don't need to have XML stuff escaped. Makes sense to wrap
output in them instead of escaping.

llvm-svn: 332116
2018-05-11 18:38:02 +00:00
Chris Matthews 7d6224d2fa Support Unsupported Tests in xunit output
We were reporting  "Unsupported" tests in xunit as passes, however since
they are not run, it make more sense to mark them as skipped. The Junit
xml standard has support for that, so lets use it.

llvm-svn: 332065
2018-05-11 00:25:43 +00:00
Chris Matthews 4855c5f717 [LIT] Move xunit tests tests into their own location, and and add failures
Failures will increase coverage.

llvm-svn: 332056
2018-05-10 22:51:28 +00:00
Chris Matthews c3f259cb6c [LIT] Add the missing file
I forgot to commit this file.

llvm-svn: 331946
2018-05-10 00:08:39 +00:00
Chris Matthews bd9ffc6243 Refactor test incase results are backwards
Looks like results can come in either way in this file.  Loosen the ordering constraints.

llvm-svn: 331945
2018-05-10 00:06:17 +00:00
Chris Matthews 069a1eb3bb [LIT] Handle xml characters in test names
Lit creates malformed xml when the test case has an & in the name.

Escape those correctly.

This also adds a test case which I will add other nasty encoding issues to in some followup commits.

llvm-svn: 331942
2018-05-09 23:48:32 +00:00
Reid Kleckner 1ca666886f Revert r330755 "[lit] Report line number for failed RUN command"
It is causing many tests to fail on Windows buildbots:
http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/10211

llvm-svn: 330848
2018-04-25 17:30:00 +00:00
Joel E. Denny 8a475307bd [lit] Report line number for failed RUN command
When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed.  For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.

When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.

To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.

Reviewed By: asmith, delcypher

Differential Revision: https://reviews.llvm.org/D44598

llvm-svn: 330755
2018-04-24 18:43:25 +00:00
Dan Liew 97a60c21d4 [lit] Remove spurious `-` in invocation of lit in
`shtest-xunit-output.py` test.

Although there is no `-` file Jeremy Morse has reported to me that it
causes problems in their setup because lit tries to find it and ends up
loading an out of tree lit configuration file.

llvm-svn: 330728
2018-04-24 15:42:00 +00:00
Dan Liew 872b8ea596 [lit] Fix a bug where UNRESOLVED tests were not handled in the XUnit
XML printer.

A test has been added that tries to comprehensively test emitting
XUnit XML output for shell tests.

Differential Revision: https://reviews.llvm.org/D45567

llvm-svn: 330409
2018-04-20 10:11:41 +00:00
Dan Liew 8ade9e75b0 Revert "[lit] Temporarily disable shtest-timeout.py on darwin"
This reverts commit 771829b640a5494ab65c810dd6b4330522bf3a33 (rr328598)

Hopefully the test will now pass on the bots.

rdar://problem/38774530

llvm-svn: 328703
2018-03-28 13:55:13 +00:00
Dan Liew 7efde3c440 [lit] Remove a timing senstive part of `shtest-timeout.py`
The `shtest-timeout.py` test was failing intermittently. It looks like
the issue is that on a resource constrained system lit is unable to run
`quick_then_slow.py` twice and print out the messages the tests expects
within the one second timeout.

The underlying issue is that the test is dependent on the performance of
the host machine is a rather fragile way. This is due to hardcoding
timeout values and having assumptions that the host machine is able to
perform a certain amount of work within the hardcoded timeout values.

We could increase the timeout values but that doesn't really fix the
underlying issue. Instead this patch removes one of fragile assumptions
in the hope that this will be enough to fix the bots.
There are other fragile assumptions in this test (e.g. `quick.py` can be
executed in less than 1 second). If the bots continue to fail we'll have
to revisit this.

rdar://problem/38774530

llvm-svn: 328702
2018-03-28 13:55:08 +00:00
Mircea Trofin 56ba71b2a7 Revert "Revert "[lit] Generalized /dev/null support on Windows.""
Summary:
This reverts commit r328596.

Checking if the arguments are strings before testing if they contain "/dev/null".

Reviewers: rnk

Reviewed By: rnk

Subscribers: delcypher, llvm-commits

Differential Revision: https://reviews.llvm.org/D44914

llvm-svn: 328603
2018-03-27 01:39:17 +00:00
Jan Korous 1e0e0b077d [lit] Temporarily disable shtest-timeout.py on darwin
Disabled until fixed in order to avoid random failures on green dragon.

rdar://problem/38774530

llvm-svn: 328598
2018-03-27 00:16:28 +00:00
Mircea Trofin 373c445c24 Revert "[lit] Generalized /dev/null support on Windows."
This reverts commit ca7fdbb974384ce5a05528b22a41d46b1cc13e92.

llvm-svn: 328596
2018-03-26 23:59:39 +00:00
Mircea Trofin 88911686c8 [lit] Generalized /dev/null support on Windows.
Generalized /dev/null remapping on Windows, and added test.

Reviewers: rnk

Reviewed By: rnk

Subscribers: amccarth, zturner, delcypher, llvm-commits

Differential Revision: https://reviews.llvm.org/D44771

llvm-svn: 328589
2018-03-26 22:41:06 +00:00
Reid Kleckner 8252892951 [lit] Implement 'cat' command for internal shell
Fixes PR36449

Patch by Chamal de Silva

Differential Revision: https://reviews.llvm.org/D43501

llvm-svn: 328563
2018-03-26 18:05:12 +00:00
Brian Homerding d5c558ff21 [lit] - Allow 1 test to report multiple micro-test results to provide support for microbenchmarks.
Summary:
These changes are to allow to a Result object to have nested Result objects in
order to support microbenchmarks. Currently lit is restricted to reporting one
result object for one test, this change provides support tests that want to
report individual timings for individual kernels.

This revision is the result of the discussions in
https://reviews.llvm.org/D32272#794759,
https://reviews.llvm.org/D37421#f8003b27 and https://reviews.llvm.org/D38496.
It is a separation of the changes purposed in https://reviews.llvm.org/D40077.

This change will enable adding LCALS (Livermore Compiler Analysis Loop Suite)
collection of loop kernels to the llvm test suite using the google benchmark
library (https://reviews.llvm.org/D43319) with tracking of individual kernel
timings.

Previously microbenchmarks had been handled by using macros to section groups
of microbenchmarks together and build many executables while still getting a
grouped timing (MultiSource/TSVC). Recently the google benchmark library was
added to the test suite and utilized with a litsupport plugin. However the
limitation of 1 test 1 result limited its use to passing a runtime option to
run only 1 microbenchmark with several hand written tests
(MicroBenchmarks/XRay). This runs the same executable many times with different
hand-written tests. I will update the litsupport plugin to utilize the new
functionality (https://reviews.llvm.org/D43316).

These changes allow lit to report micro test results if desired in order to get
many precise timing results from 1 run of 1 test executable.


Reviewers: MatzeB, hfinkel, rengolin, delcypher

Differential Revision: https://reviews.llvm.org/D43314

llvm-svn: 327422
2018-03-13 16:37:59 +00:00
Lei Huang 9db7bd5cab Losen time contraint to accommodate system loads
llvm-svn: 326359
2018-02-28 20:05:24 +00:00
Max Moroz 975eacada5 [lit] Implement "-r" option for builtin "diff" command + a test using that.
Summary:
That would allow to recursively compare directories in tests using
"diff -r" on Windows in a similar way as it can be done on Linux or Mac.

Reviewers: zturner, morehouse, vsk

Reviewed By: zturner

Subscribers: kcc, llvm-commits

Differential Revision: https://reviews.llvm.org/D41776

llvm-svn: 322102
2018-01-09 18:23:34 +00:00
Ying Yi cfb08e9e6b [lit] Implement non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands internally
Summary:
The internal shell already supports 'cd', ‘export’ and ‘echo’ commands.
This patch adds implementation of non-pipelined ‘mkdir’, ‘diff’ and ‘rm’
commands as the internal shell builtins.

Reviewed by: Zachary Turner, Reid Kleckner

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D39567

llvm-svn: 319528
2017-12-01 09:54:27 +00:00
Greg Bedwell 34a83f0faf Make check-lit tests respect LLVM_LIT_TOOLS_DIR
Differential Revision: https://reviews.llvm.org/D40520

llvm-svn: 319329
2017-11-29 18:05:26 +00:00
Ying Yi a0903c6e5d Reverted rL318911 since it broke the sanitizer-windows.
llvm-svn: 318914
2017-11-23 13:23:21 +00:00
Ying Yi 989c9e75a6 [lit] Implement non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands internally
Summary:
The internal shell already supports 'cd', ‘export’ and ‘echo’ commands. 
This patch adds implementation of non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ 
commands as the internal shell builtins.

Reviewers: Zachary Turner, Reid Kleckner

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D39567

llvm-svn: 318911
2017-11-23 12:48:41 +00:00
Jordan Rose e55d9a1403 Revert "lit.py: Add new %{shared_output(LABEL)} substitution"
This reverts r315697 and my ill-fated attempts to fix it on Windows.
I'll try again when I get access to a Windows machine.

llvm-svn: 315793
2017-10-14 04:01:27 +00:00
Jordan Rose a33a17605d lit.py: Give up and disable the new shared-output.py test on Windows
"No such file or directory: C:\\...\\tests\\Output\\shared-output.py.tmp/Output/Shared/SHARED.tmp"

And yet other forward-slashes don't seem to be causing the same
problem. I'll see if I can get ahold of a Windows machine to poke at
this directly later.

llvm-svn: 315792
2017-10-14 03:33:46 +00:00
Jordan Rose fddfb5ad4c lit.py: Previous test fix was a red herring; backslashes are escapes
I don't have access to a Windows machine at the moment, so if this
doesn't fix it I'll just revert for now.

llvm-svn: 315782
2017-10-14 00:49:19 +00:00
Jordan Rose aafbc1c4db lit.py: Fix new test harder for systems that don't use / as os.path.sep
I didn't think about '%{inputs}' having the same problem. This one
should be a fully Windows path name.

llvm-svn: 315779
2017-10-14 00:26:48 +00:00
Jordan Rose f4fd5938e7 lit.py: Fix new test for systems that don't use / as os.path.sep
llvm-svn: 315773
2017-10-13 22:46:40 +00:00
Jordan Rose 53be3224b2 lit.py: Add new %{shared_output(LABEL)} substitution
This refers to a temporary path that can be shared across all tests,
identified by a particular label. This can be used for things like
caches.

At the moment, the character set for the LABEL is limited to C
identifier characters, plus '-', '+', '=', and '.'. This is the same
set of characters currently allowed in REQUIRES clause identifiers.

llvm-svn: 315697
2017-10-13 16:12:23 +00:00
Michal Gorny d6a4c79b14 [lit] Fix running lit tests in unconfigured source dir
Fix llvm_tools_dir attribute access not to fail when the variable is not
present. This directory is not really necessary to run lit tests,
and the code already accounts for it being None.

The reference was added in r313407, and it breaks the stand-alone lit
package in Gentoo.

Differential Revision: https://reviews.llvm.org/D38442

llvm-svn: 314620
2017-10-01 07:13:25 +00:00
Zachary Turner 1ca789bdba [lit] Actually do normalize the case of files in the config map.
This has gone back and forth, but it seems this is necessary
after all.  realpath is not sufficient because if you have a
file named 'C:\foo.txt', then both realpath('c:\foo.txt') and
realpath(C:\foo.txt') return the string that was passed to them
exactly as is, meaning the case of the drive-letter won't match.

The problem before was not that we were normalizing the case of
items going into the config map, but rather that we were
normalizing the case of something we needed to print.  The value
that is used to key on the config map should never be printed.

llvm-svn: 313918
2017-09-21 21:27:11 +00:00
Zachary Turner 43bcf226c1 [lit] Don't norm case when inserting into the config map.
This makes all paths lowercase on Windows, which seemed like a
good idea at the time, but it means that tests can't properly
use FileCheck to match expected path names.

llvm-svn: 313889
2017-09-21 17:02:08 +00:00