When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:
```
# RUN: program | diff file -
# RUN: not diff file1 file2 | FileCheck %s
```
Such cases exist now, in `clang/test/Analysis` for example. We are
preparing patches to ensure lit's internal `diff` is called in such
cases, which will then fail because lit's internal `diff` cannot
currently be used in pipelines and doesn't recognize `-` as a
command-line option.
To enable pipelines, this patch moves lit's `diff` implementation into
an out-of-process script, similar to lit's `cat` implementation. A
follow-up patch will implement `-` to mean stdin.
Reviewed By: probinson, stella.stamenova
Differential Revision: https://reviews.llvm.org/D66574
llvm-svn: 372035
Without this patch, failing to provide a subcommand to lit's internal
`env` results in either a python `IndexError` or an attempt to execute
the final `env` argument, such as `FOO=1`, as a command. This patch
diagnoses those cases with a more helpful message.
Reviewed By: stella.stamenova
Differential Revision: https://reviews.llvm.org/D66482
llvm-svn: 369620
This patch extends lit's test suite to check that lit's internal shell
doesn't accidentally execute internal commands as external commands.
It does so by putting fake failing versions of those commands in
`PATH` while the entire lit test suite is running. Without the fixes
in D65697 but with its tests, this approach catches accidental
external `env` calls.
Reviewed By: probinson
Differential Revision: https://reviews.llvm.org/D66293
llvm-svn: 369309
Without this patch, the internal `env` command removes `env` and its
args from the command line while parsing it. This patch modifies a
copy instead so that the original command line is printed.
Reviewed By: stella.stamenova, rnk
Differential Revision: https://reviews.llvm.org/D65624
llvm-svn: 367752
Put the main test script in the right directory, and fix a python bug
in a local script.
Reviewed By: stella.stamenova
Differential Revision: https://reviews.llvm.org/D65623
llvm-svn: 367751
Summary:
This change updates the lit.cfg file to use llvm_config when it is available, but when it is not, it directly modifies the config object. This makes it possible to run the lit tests standalone without having built llvm (as long as the correct binaries are present in the path such as FileCheck and not).
Because the lit tests don't take a hard dependency on llvm_config, some features such as system-windows have to have definitions in lit's cfg file as well. This is a potential issue as the os features sometimes change names (for example, we went from windows to system-windows, etc.). This can cause drift between lit's tests and the rest of the llvm tests.
Reviewers: probinson, mgorny
Reviewed By: mgorny
Subscribers: delcypher, llvm-commits, asmith
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D65674
llvm-svn: 367730
Set environment variables to empty values rather than attempting
to unset them via 'env -u', in order to fix NetBSD test regression
caused by r366980. POSIX does not guarantee that env(1) supports '-u'
option, and indeed NetBSD env(1) does not support it.
Differential Revision: https://reviews.llvm.org/D65335
llvm-svn: 367123
lit's test suite calls lit multiple times for various sample test
suites. `FILECHECK_OPTS` is safe for FileCheck calls in lit's test
suite. It's not safe for FileCheck calls in the sample test suites,
whose output affects the results of lit's test suite.
Without this patch, only one such sample test suite is protected from
`FILECHECK_OPTS`, and I admit I haven't discovered other cases for
which I can produce false failures using `FILECHECK_OPTS`. However,
it's hard to predict the future, especially false passes. Thus, this
patch protects all existing and future sample test suites from
`FILECHECK_OPTS` (and the deprecated
`FILECHECK_DUMP_INPUT_ON_FAILURE`).
Reviewed By: probinson
Differential Revision: https://reviews.llvm.org/D65156
llvm-svn: 366980
Summary:
On AIX psutil can run into problems with permissions to read the process
tree, which causes problems for python timeout tests which need to kill off
a test and it's children.
This patch adds a workaround by invoking shell via subprocess and using a
platform specific option to ps to list all the descendant processes so we can
kill them. We add some checks so lit can tell whether timeout tests are
supported with out exposing whether we are utilizing the psutil
implementation or the alternative.
Reviewers: hubert.reinterpretcast, andusy, davide, delcypher
Reviewed By: delcypher
Subscribers: davide, delcypher, christof, lldb-commits, libcxx-commits, llvm-commits
Tags: #lldb, #libc, #llvm
Differential Revision: https://reviews.llvm.org/D64251
llvm-svn: 366912
Similar to `FILECHECK_OPTS` for FileCheck, `LIT_OPTS` makes it easy to
adjust lit behavior when running the test suite via ninja. For
example:
```
$ LIT_OPTS='--time-tests -vv --filter=threadprivate' \
ninja check-clang-openmp
```
Reviewed By: probinson
Differential Revision: https://reviews.llvm.org/D64135
llvm-svn: 365313
Ensure that the bash script written by lit TestRunner is open with UTF-8
encoding when using Python 3. Otherwise, attempt to write non-ASCII
characters causes UnicodeEncodeError. This happened e.g. with
the following LLD test:
UNRESOLVED: lld :: ELF/format-binary-non-ascii.s (657 of 2119)
******************** TEST 'lld :: ELF/format-binary-non-ascii.s' FAILED ********************
Exception during script execution:
Traceback (most recent call last):
File "/home/mgorny/llvm-project/llvm/utils/lit/lit/worker.py", line 63, in _execute_test
result = test.config.test_format.execute(test, lit_config)
File "/home/mgorny/llvm-project/llvm/utils/lit/lit/formats/shtest.py", line 25, in execute
self.execute_external)
File "/home/mgorny/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1644, in executeShTest
res = _runShTest(test, litConfig, useExternalSh, script, tmpBase)
File "/home/mgorny/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1590, in _runShTest
res = executeScript(test, litConfig, tmpBase, script, execdir)
File "/home/mgorny/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1157, in executeScript
f.write('{ ' + '; } &&\n{ '.join(commands) + '; }')
UnicodeEncodeError: 'ascii' codec can't encode character '\xa3' in position 274: ordinal not in range(128)
Differential Revision: https://reviews.llvm.org/D63254
llvm-svn: 363388
Summary: This also normalizes the config feature that represents the windows platform to "system-windows" as opposed to having both "windows" and "system-windows"
Reviewers: asmith, probinson
Subscribers: delcypher, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D61798
llvm-svn: 361998
Summary:
Various tests in the `lit` testing suite expect specific return codes
and forms of diagnostic message from utility programs. As per
POSIX.1-2017 XCU Section 1.4, Utility Description Defaults, "[the]
format of diagnostic messages for most utilities is unspecified".
The STDERR subsections of the `cat` and `wc` utilities merely indicate
that "[the] standard error shall be used only for diagnostic messages".
The corresponding EXIT STATUS subsections merely indicate, with regard
to errors, an exit value of >0.
The affected tests are updated to accept the applicable diagnostic
message as produced by the utilities on AIX. The exit value is
normalized using `not` as necessary.
Reviewers: xingxue, sfertile, jasonliu
Reviewed By: xingxue
Subscribers: delcypher, jsji, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D60553
llvm-svn: 359690
This enables lit to work with unicode file names via mkdir, rm, and redirection.
Lit still uses utf-8 internally, but converts to utf-16 on Windows, or just utf-8
bytes on everything else.
Committed on behalf of Jason Mittertreiner
Differential Revision: https://reviews.llvm.org/D56754
llvm-svn: 355122
Check that we do not crash if a parallelism group is explicitly set to
None. Permits usage of the following pattern.
[lit.common.cfg]
lit_config.parallelism_groups['my_group'] = None
if <condition>:
lit_config.parallelism_groups['my_group'] = 3
[project/lit.cfg]
config.parallelism_group = 'my_group'
Reviewers: rnk
Differential Revision: https://reviews.llvm.org/D58305
llvm-svn: 354912
From the docs: `class LitTestCase(unittest.TestCase)`
LitTestCase is an adaptor for providing a 'unittest' compatible
interface to 'lit' tests so that we can run lit tests with standard
python test runners.
It does not seem to be used anywhere.
Differential Revision: https://reviews.llvm.org/D58264
llvm-svn: 354188
Summary:
Automatically upgrade debugging experience (single process, no thread
pool) when:
1) we only run a single test
2) user specifies `-j1`
Details:
Fix `--max-failures` in single process mode. Option did not have an
effect in single process mode.
Add display feedback for single process mode. Adapted test.
Improve argument checking (require positive integers).
`--single-process` is now essentially an alias for `-j1`. Should we
remove it?
Reviewers: rnk
Differential Revision: https://reviews.llvm.org/D58249
llvm-svn: 354068
If a user has PYTHONPATH set in the environment, append new entries to
it rather than blindly setting PYTHONPATH to a fixed string. This
allows tests to, for example, find psutil if it is in
PYTHONPATH. Without this change, lit will detect psutil but then
various tests will fail because PYTHONPATH has been overwritten and
psutil cannot be found.
llvm-svn: 350536
Make sure all print statements are compatible with Python 2 and Python3 using
the `from __future__ import print_function` statement.
Differential Revision: https://reviews.llvm.org/D56249
llvm-svn: 350307
Summary: This is the only test that is still failing on Windows - or rather, it is expected to fail on the bots, but passes on the new bot that we're preparing causing a failure, so I'm going to disable it. Since the test has rarely, if ever, passed on the bots, this should have the same effect and it will unblock the creation of the new bot.
Reviewers: asmith, delcypher, zturner
Subscribers: stella.stamenova, llvm-commits
Differential Revision: https://reviews.llvm.org/D51871
llvm-svn: 341856
Summary:
Right now this test is failing on the builtbots on Windows but we have a very similar setup where the test passes. The test is meant to test that specifying a timeout works correctly by running an infnite loop and having it timeout - on the buildbot, the infinite loop doesn't actually execute. This change runs all of the tests in the set using an internal shell rather than an external shell. I expect this will make the test pass which means that either the way the external shell is invoked or the external shell setup on the buildbots is not correct. Regardless of whether the test passes with this change, we'll need to undo this change and have a real fix.
@gkistanova was able to get logs from the buildbot to rule out a number of theories as to why this test is failing, but they didn't have enough information to confirm exactly what the issue is. The purpose of this change is to narrow it down, but if someone has a local repro and can aid in debugging, that would make it much speedier (and less prone to making the bots fail).
Reviewers: gkistanova, asmith, zturner, modocache, rnk, delcypher
Reviewed By: rnk
Subscribers: delcypher, llvm-commits, gkistanova
Differential Revision: https://reviews.llvm.org/D51326
llvm-svn: 340840
This test passes on Windows when using Python 3 but fails when using Python 2, so it needs more investigation before it can be enabled as the bots use Python 2.
llvm-svn: 339184
Summary:
In Python2 'unicode' is a distinct type from 'str', but in Python3 'unicode' does not exist and instead all 'str' objects are Unicode string. This change updates the logic in the test logging for lit to correctly process each of the types, and more importantly, to not just fail in Python3.
This change also reverses the use of quotes in several of the cfg files. By using '""' we are guaranteeing that the resulting path will work correctly on Windows while "''" only works correctly sometimes. This also fixes one of the failing tests.
Reviewers: asmith, zturner
Subscribers: stella.stamenova, delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D50397
llvm-svn: 339179
Summary:
The problem here is that on windows double quotes are used for paths (usually) while single quotes are not. This is not generally a problem for the tests because the lit infrastructure tends to treat both the same. One (and possibly only) exception is when some tests are run in an external shell such as some of the shtest-format tests. In this case on windows the path to python was not created correctly because it had single quotes and the test failed.
This same test is already failing with python 3 which is why our testing missed the new failure. This patch will take care of the immediate failure with python 2 and I'll send a follow up for the python 3 failure.
Reviewers: asmith, zturner
Subscribers: delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D50373
llvm-svn: 339091
Summary:
The issue with the python path is that the path to python on Windows can contain spaces. To make the tests always work, the path to python needs to be surrounded by quotes.
This change updates several configuration files which specify the path to python as a substitution and also remove quotes from existing tests.
Reviewers: asmith, zturner, alexshap, jakehehrlich
Reviewed By: zturner, alexshap, jakehehrlich
Subscribers: mehdi_amini, nemanjai, eraman, kbarton, jakehehrlich, steven_wu, dexonsmith, stella.stamenova, delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D50206
llvm-svn: 339073
These two tests are operating on the same test suite, which causes
them to be racy about writing temporary files and can cause spurious
failures. Merge them into one test to avoid the issue.
llvm-svn: 337718
The test was failing on Windows machines which had bash.exe on PATH (but
not in the so called lit tools dir, containing cmp.exe, grep.exe etc.).
The problem was that the outer lit invocation would load LLVMConfig
from utils/lit/lit/llvm/config.py, which looks up the tools path with
getToolsPath(). That has a surprising side effect of also setting
bashPath, in our case setting it to empty.
The outer lit invocation would thus configure the pdbg0 and pdbg1
substitutions based on not running with bash.
But the inner lit invocation would not load LLVMConfig, so bash
would be found on PATH, that would be used as external shell,
and so the output wouldn't match pdbg0 and pdbg1.
It seems weird to me that getBashPath() will return different results
depending on whether getToolsPath() has been called before, but I
also don't know how to fix it properly.
This commit just relaxes the test case, because there doesn't seem
to be much point in testing for the exact syntax of the run file
as long as it works.
(See https://crbug.com/850023)
llvm-svn: 334100
This fixes projects/compiler-rt/test/fuzzer/sigusr.test, which was
broken by r333614. The trouble was that "&&" changes the command for
which "$!" gives the pid.
llvm-svn: 333620
(Relands r333584, reverted in 333592.)
When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.
When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.
To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.
To support reporting RUN line numbers in the case of windows cmd.exe
as the external shell, this patch extends -vv to set "echo on" instead
of "echo off" in bat files. (Support for windows cmd.exe as a lit
external shell will likely be dropped later, but I found out too
late.)
Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk
Differential Revision: https://reviews.llvm.org/D44598
llvm-svn: 333614
(Relands r330755 (reverted in r330848) with fix for PR37239.)
When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.
When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.
To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.
To support reporting RUN line numbers in the case of windows cmd.exe
as the external shell, this patch extends -vv to set "echo on" instead
of "echo off" in bat files. (Support for windows cmd.exe as a lit
external shell will likely be dropped later, but I found out too
late.)
Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk
Differential Revision: https://reviews.llvm.org/D44598
llvm-svn: 333584
We were making malformed XML on tests with ' in the name. Switch to
using saxutils to set all of our attributes, so it can handle quotes
etc correctly.
llvm-svn: 333249
larger timeout value. This really isn't very good because it will
still be susceptible to machine performance.
While we are here also fix a bug in validation of
`maxIndividualTestTime` where previously it wasn't checked if the
type was an int.
rdar://problem/40221572
llvm-svn: 332987
The program used to be used in `quick_then_slow.py` but that was
removed in r328702. The tests always run `slow.py` on its own but
this doesn't really test additional code so we'll just drop running
`slow.py` so the tests run faster.
rdar://problem/40221572
llvm-svn: 332986
If the system is under heavy load 1 second might not be long enough
for it to produce output which could lead to spurious test failures.
What matters is that the right test cases reach a timeout.
rdar://problem/40221572
llvm-svn: 332985
Summary:
This sequence ends the CDATA block so any characters after that are no
longer escaped. This can be fixed by replacing "]]>" with "]]]]><
Joel E. Denny