Summary:
This issue was found when running the clang unit test on Windows. Python 3.x cannot open some of the files that the tests are using with a simple open because of their encoding. Python 2.7+ and Python 3.x both support io.open which allows for an encoding to be specified.
This change will determine whether two files being compared should be opened (and then compared) as text or binary and whether to use utf-8 or the default encoding before proceeding with a line-by-line comparison.
Patch by Stella Stamenova!
Reviewers: zturner, llvm-commits, rnk, MaggieYi
Reviewed By: zturner
Subscribers: asmith, MatzeB, stella.stamenova, delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D43165
llvm-svn: 329012
This reverts commit 771829b640a5494ab65c810dd6b4330522bf3a33 (rr328598)
Hopefully the test will now pass on the bots.
rdar://problem/38774530
llvm-svn: 328703
The `shtest-timeout.py` test was failing intermittently. It looks like
the issue is that on a resource constrained system lit is unable to run
`quick_then_slow.py` twice and print out the messages the tests expects
within the one second timeout.
The underlying issue is that the test is dependent on the performance of
the host machine is a rather fragile way. This is due to hardcoding
timeout values and having assumptions that the host machine is able to
perform a certain amount of work within the hardcoded timeout values.
We could increase the timeout values but that doesn't really fix the
underlying issue. Instead this patch removes one of fragile assumptions
in the hope that this will be enough to fix the bots.
There are other fragile assumptions in this test (e.g. `quick.py` can be
executed in less than 1 second). If the bots continue to fail we'll have
to revisit this.
rdar://problem/38774530
llvm-svn: 328702
Summary:
This reverts commit r328596.
Checking if the arguments are strings before testing if they contain "/dev/null".
Reviewers: rnk
Reviewed By: rnk
Subscribers: delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D44914
llvm-svn: 328603
Summary:
These changes are to allow to a Result object to have nested Result objects in
order to support microbenchmarks. Currently lit is restricted to reporting one
result object for one test, this change provides support tests that want to
report individual timings for individual kernels.
This revision is the result of the discussions in
https://reviews.llvm.org/D32272#794759,
https://reviews.llvm.org/D37421#f8003b27 and https://reviews.llvm.org/D38496.
It is a separation of the changes purposed in https://reviews.llvm.org/D40077.
This change will enable adding LCALS (Livermore Compiler Analysis Loop Suite)
collection of loop kernels to the llvm test suite using the google benchmark
library (https://reviews.llvm.org/D43319) with tracking of individual kernel
timings.
Previously microbenchmarks had been handled by using macros to section groups
of microbenchmarks together and build many executables while still getting a
grouped timing (MultiSource/TSVC). Recently the google benchmark library was
added to the test suite and utilized with a litsupport plugin. However the
limitation of 1 test 1 result limited its use to passing a runtime option to
run only 1 microbenchmark with several hand written tests
(MicroBenchmarks/XRay). This runs the same executable many times with different
hand-written tests. I will update the litsupport plugin to utilize the new
functionality (https://reviews.llvm.org/D43316).
These changes allow lit to report micro test results if desired in order to get
many precise timing results from 1 run of 1 test executable.
Reviewers: MatzeB, hfinkel, rengolin, delcypher
Differential Revision: https://reviews.llvm.org/D43314
llvm-svn: 327422
On Windows, if the substitution contains a back reference, it would
removed due to the replacement of the escape character in lit. Create a
helper class to avoid this which will simply ignore the replacement and
mark the substitution as having capture groups being referenced.
llvm-svn: 327082
We are running lld tests with "--full-shutdown" option because we don't
want to call _exit() in lld if it is running tests. Regular shutdown
is needed for leak sanitizer.
This patch changes the way how we tell lld that it is running tests.
Now "--full-shutdown" is removed, and LLD_IN_TEST environment variable
is used instead.
This patch enables full shutdown on all ports, e.g. ELF, COFF and wasm.
Previously, we enabled it only for ELF.
Differential Revision: https://reviews.llvm.org/D43410
llvm-svn: 325413
Allow CLANG environment variable be copied into the testing configuration
and proper support testing with a custom path to the clang executable.
Signed-off-by: Vladimir Vereschaka <vvereschaka@accesssoftek.com>
llvm-svn: 324706
Summary:
That would allow to recursively compare directories in tests using
"diff -r" on Windows in a similar way as it can be done on Linux or Mac.
Reviewers: zturner, morehouse, vsk
Reviewed By: zturner
Subscribers: kcc, llvm-commits
Differential Revision: https://reviews.llvm.org/D41776
llvm-svn: 322102
The variable named `minor` was actually pointing to the patch part of
the version. While I was changing this I also made the check for Apple
clang more robust by checking both patch and minor rather than just
minor.
llvm-svn: 319656
The latest clang that ships with Xcode (clang 900 or 9.0.0) does not
support LSan. This fixes the lit configuration to reflect that.
Differential revision: https://reviews.llvm.org/D40672
llvm-svn: 319530
This is still breaking greendragon.
At this point I give up until someone can fix the greendragon
bots, and I will probably abandon this effort in favor of using
a private github repository.
llvm-svn: 318722
This was reverted due to the tests being run twice on some
build bots. Each run had a slightly different configuration
due to the way in which it was being invoked. This fixes
the problem (albeit in a somewhat hacky way). Hopefully in
the future we can get rid of the workflow of running
debuginfo-tests as part of clang, and then this hack can
go away.
llvm-svn: 318697
Summary:
Currently, LIT configures the LLVM binary path before the Clang binary path. However this breaks testing out-of-tree Clang builds (where the LLVM binary path includes a copy of Clang).
This patch reverses the order of the paths when looking for Clang, putting the Clang binary directory first.
Reviewers: zturner, beanz, chapuni, modocache, EricWF
Reviewed By: EricWF
Subscribers: mgorny, cfe-commits, llvm-commits
Differential Revision: https://reviews.llvm.org/D40217
llvm-svn: 318607
This is still broken because it causes certain tests to be
run twice with slightly different configurations, which is
wrong in some cases.
You can observe this by running:
ninja -nv check-all | grep debuginfo-tests
And seeing that it passes clang/test and clang/test/debuginfo-tests
to lit, which causes it to run debuginfo-tests twice. The fix is
going to involve either:
a) figuring out that we're running in this "deprecated" configuration,
and then deleting the clang/test/debuginfo-tests path, which should
cause it to behave identically to before, or:
b) make lit smart enough that it doesn't descend into a sub-suite if
that sub-suite already has a lit.cfg file.
llvm-svn: 318486
This was reverted due to some failures on specific darwin buildbots,
the issue being that the new lit configuration was not setting the
SDKROOT environment variable. We've tested a fix locally and confirmed
that it works, so this patch resubmits everything with the fix
applied.
llvm-svn: 318435
This reverts the aforementioned patch and 2 subsequent follow-ups,
as some buildbots are still failing 2 tests because of it.
Investigation is ongoing into the cause of the failures.
llvm-svn: 318112
Previously, debuginfo-tests was expected to be checked out into
clang/test and then the tests would automatically run as part of
check-clang. This is not a standard workflow for handling
external projects, and it brings with it some serious drawbacks
such as the inability to depend on things other than clang, which
we will need going forward.
The goal of this patch is to migrate towards a more standard
workflow. To ease the transition for build bot maintainers,
this patch tries not to break the existing workflow, but instead
simply deprecate it to give maintainers a chance to update
the build infrastructure.
Differential Revision: https://reviews.llvm.org/D39605
llvm-svn: 317925
I need a test that only runs in a reasonable amount of time on systems
that have sparse files. The broadest class of systems that support
sparse files are linux systems. So restricting my test to linux systems
should suffice. This change adds the system-linux feature to llvm-lit so
that it can be required.
Differential Revision: https://reviews.llvm.org/D39482
llvm-svn: 317055
The new scheme should match the normalization of embedded paths in
linkrepro tar files.
Differential Revision: https://reviews.llvm.org/D39023
llvm-svn: 316044
"No such file or directory: C:\\...\\tests\\Output\\shared-output.py.tmp/Output/Shared/SHARED.tmp"
And yet other forward-slashes don't seem to be causing the same
problem. I'll see if I can get ahold of a Windows machine to poke at
this directly later.
llvm-svn: 315792
This refers to a temporary path that can be shared across all tests,
identified by a particular label. This can be used for things like
caches.
At the moment, the character set for the LABEL is limited to C
identifier characters, plus '-', '+', '=', and '.'. This is the same
set of characters currently allowed in REQUIRES clause identifiers.
llvm-svn: 315697
This paves the way for other projects which might /use/ clang or
lld but not necessarily need to the full set of functionality
available to clang and lld tests to be able to have a basic set
of substitutions that allow a project to run the clang or lld
executables.
llvm-svn: 315627
This addresses two sources of inconsistency in test configuration
files.
1. Substitution boundaries. Previously you would specify a
substitution, such as 'lli', and then additionally a set
of characters that should fail to match before and after
the tool. This was used, for example, so that matches that
are parts of full paths would not be replaced. But not all
tools did this, and those that did would often re-invent
the set of characters themselves, leading to inconsistency.
Now, every tool substitution defaults to using a sane set
of reasonable defaults and you have to explicitly opt out
of it. This actually fixed a few latent bugs that were
never being surfaced, but only on accident.
2. There was no standard way for the system to decide how to
locate a tool. Sometimes you have an explicit path, sometimes
we would search for it and build up a path ourselves, and
sometimes we would build up a full command line. Furthermore,
there was no standardized way to handle missing tools. Do we
warn, fail, ignore, etc? All of this is now encapsulated in
the ToolSubst class. You either specify an exact command to
run, or an instance of FindTool('<tool-name>') and everything
else just works. Furthermore, you can specify an action to
take if the tool cannot be resolved.
Differential Revision: https://reviews.llvm.org/D38565
llvm-svn: 315085
Summary:
normpath() was being called on an empty string and appended to
the environment variable in the case where the environment variable
was unset. This led to ":." being appended to the path, since
normpath() of an empty string is '.', presumably to represent cwd.
Reviewers: zturner, sqlbyme, modocache
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D38542
llvm-svn: 314915
Fix llvm_tools_dir attribute access not to fail when the variable is not
present. This directory is not really necessary to run lit tests,
and the code already accounts for it being None.
The reference was added in r313407, and it breaks the stand-alone lit
package in Gentoo.
Differential Revision: https://reviews.llvm.org/D38442
llvm-svn: 314620
Summary:
Also disables leak checking on lto tests, due to many leaks reported
in the system's ld64.
Reviewers: kcc, pcc, bogner, kubamracek
Subscribers: mehdi_amini, llvm-commits
Differential Revision: https://reviews.llvm.org/D37781
llvm-svn: 314535
There were two issues, one Python 3 specific related to Unicode,
and another which is that the tool substitution for lld no longer
rejected matches where a / preceded the tool name.
llvm-svn: 313928
debuginfo-tests has need to reuse a lot of common configuration
from clang and lld, and in general it seems like all of the
projects which are tightly coupled (e.g. lld, clang, llvm, lldb,
etc) can benefit from knowing about one other. For example,
lldb needs to know various things about how to run clang in its
test suite. Since there's a lot of common substitutions and
operations that need to be shared among projects, sinking this
up into LLVM makes sense.
In addition, this patch introduces a function add_tool_substitution
which handles all the dirty intricacies of matching tool names
which was previously copied around the various config files. This
is now a simple straightforward interface which is hard to mess
up.
Differential Revision: https://reviews.llvm.org/D37944
llvm-svn: 313919
This has gone back and forth, but it seems this is necessary
after all. realpath is not sufficient because if you have a
file named 'C:\foo.txt', then both realpath('c:\foo.txt') and
realpath(C:\foo.txt') return the string that was passed to them
exactly as is, meaning the case of the drive-letter won't match.
The problem before was not that we were normalizing the case of
items going into the config map, but rather that we were
normalizing the case of something we needed to print. The value
that is used to key on the config map should never be printed.
llvm-svn: 313918
This makes all paths lowercase on Windows, which seemed like a
good idea at the time, but it means that tests can't properly
use FileCheck to match expected path names.
llvm-svn: 313889
Config map is not exposed through the command line, so testing this
is somewhat tricky. But basically we need a test that if a custom
driver builds a config map and passes it to main, it gets respected.
A config map allows config files in the source tree to be mapped
to alternate config files in the build tree. This particular test
works by having two config files in separate directories, and
setting up a config map to have that redirects A/lit.site.cfg
to B/altconfig. Then, we print a message in A/lit.site.cfg
and B/altconfig and check that we do see the output from B
but don't see the output from A. Additionally we test that
the test suite specified by A's config map is properly discovered.
Differential Revision: https://reviews.llvm.org/D38105
llvm-svn: 313887
Summary:
This appears to break some bots, when getToolsPath fails to find some or
all of the tools (for example, an incomplete GnuWin32 installation).
Reviewers: zturner, modocache
Subscribers: sanjoy, llvm-commits
Differential Revision: https://reviews.llvm.org/D38115
llvm-svn: 313854
Many editors and Python-related diagnostics tools such as
debuggers break or fail in mysterious ways when python files
don't end in .py. This is especially true on Windows, but
still exists on other platforms. I don't want to be too heavy
handed in changing everything across the board, but I do want
to at least *allow* lit configs to have .py extensions. This
patch makes the discovery process first look for a config file
with a .py extension, and if one is not found, then looks for
a config file using the old method. So for existing users, there
should be no functional change.
Differential Revision: https://reviews.llvm.org/D37838
llvm-svn: 313849
Bug pointed out by EricWF. This would construct a path where
items would be added in the wrong order, potentially leading
to using the wrong tools for testing.
llvm-svn: 313765
Many svn-based buildbots seem to be getting stuck continually
in tree conflicts due to the output of pyc files. I'm disabling
these as a temporary measure in an attempt to get everything
stable again.
I'll try to remove this code once I understand the problem
better.
llvm-svn: 313698
Since the path a user specifies to the llvm-lit script might be
different than the source tree they built from (since they could
be behind different symlinks), we need to use realpath to make
sure that path comparisons work as expected.
Even better would be to use a custom dictionary comparison with
actual file equivalence comparison semantics, but this is the
least friction to unbreak things for now.
llvm-svn: 313594
It doesn't make sense to me why these bots are failing as the
traceback does not agree with the source code. It's possible
something is stale or there is some other mysterious error,
but in any case hopefully this fixes it.
llvm-svn: 313469
A few tests were manually constructing a LitConfig object, since
I added a new argument to it this was triggering some failures
I didn't detect. `ninja check-lit` passes now.
llvm-svn: 313461
This is helpful for debugging test failures since it removes
the multiprocessing pool from the picture. This will obviously
slow down the test suite by a few orders of magnitude, so it
should only be used for debugging specific failures.
llvm-svn: 313460
It looks like this is going to be non-trivial to get working
in both Py2 and Py3, so for now I'm reverting until I have time
to fully test it under Python 3.
llvm-svn: 313429
This is a resubmission of r313270. It broke standalone builds of
compiler-rt because we were not correctly generating the llvm-lit
script in the standalone build directory.
The fixes incorporated here attempt to find llvm/utils/llvm-lit
from the source tree returned by llvm-config. If present, it
will generate llvm-lit into the output directory. Regardless,
the user can specify -DLLVM_EXTERNAL_LIT to point to a specific
lit.py on their file system. This supports the use case of
someone installing lit via a package manager. If it cannot find
a source tree, and -DLLVM_EXTERNAL_LIT is either unspecified or
invalid, then we print a warning that tests will not be able
to run.
Differential Revision: https://reviews.llvm.org/D37756
llvm-svn: 313407
This patch is still breaking several multi-stage compiler-rt bots.
I already know what the fix is, but I want to get the bots green
for now and then try re-applying in the morning.
llvm-svn: 313335
To further reduce duplicate code, this patch introduces a module
that configs can simply import and get access to a lot of useful
functionality such as setting up paths, adding features that are
useful across all projects, and other utility-type functions.
For now this only updates llvm's suite to use this new library,
but subsequent patches will update other projects.
Differential Revision: https://reviews.llvm.org/D37778
llvm-svn: 313325
This patch simplifies LLVM's lit infrastructure by enforcing an ordering
that a site config is always run before a source-tree config.
A significant amount of the complexity from lit config files arises from
the fact that inside of a source-tree config file, we don't yet know if
the site config has been run. However it is *always* required to run
a site config first, because it passes various variables down through
CMake that the main config depends on. As a result, every config
file has to do a bunch of magic to try to reverse-engineer the location
of the site config file if they detect (heuristically) that the site
config file has not yet been run.
This patch solves the problem by emitting a mapping from source tree
config file to binary tree site config file in llvm-lit.py. Then, during
discovery when we find a config file, we check to see if we have a
target mapping for it, and if so we use that instead.
This mechanism is generic enough that it does not affect external users
of lit. They will just not have a config mapping defined, and everything
will work as normal.
On the other hand, for us it allows us to make many simplifications:
* We are guaranteed that a site config will be executed first
* Inside of a main config, we no longer have to assume that attributes
might not be present and use getattr everywhere.
* We no longer have to pass parameters such as --param llvm_site_config=<path>
on the command line.
* It is future-proof, meaning you don't have to edit llvm-lit.in to add
support for new projects.
* All of the duplicated logic of trying various fallback mechanisms of
finding a site config from the main config are now gone.
One potentially noteworthy thing that was required to implement this
change is that whereas the ninja check targets previously used the first
method to spawn lit, they now use the second. In particular, you can no
longer run lit.py against the source tree while specifying the various
`foo_site_config=<path>` parameters. Instead, you need to run
llvm-lit.py.
Differential Revision: https://reviews.llvm.org/D37756
llvm-svn: 313270
Use os.path.normpath instead of realpath to collapse '..' and '.' path
components. Use realpath when caching search results about a path for
good measure.
I considered rigging up a test involving symlinks for this, but I doubt
I can check a symlink into SVN. The test would have to conditionally
create a symlink at runtime if the host OS supports it. This sounds too
fragile and complicated to me to be worth it.
llvm-svn: 312254
This preserves symlinks in paths, so that someone can symlink more tests
into a larger test suite. For example, debuginfo-tests is currently
designed to be checked out into clang/test. With this change, it can be
symlinked into place instead, which works better with the monorepo.
llvm-svn: 312250
It was marked as unsupported on Windows in r311230 because on some Win10
machines it failed or caused hang. The problem was that on these machines
system bash (C:\Windows\System32\bash.exe) was used which requires paths to be
passed like '/mnt/c/path/to/my/script' instead of 'C:\path\to\my\script'.
TODO: we should make lit detect if system bash is used instead of msys and set
appropriate path format.
llvm-svn: 311558
This is an updated version of https://reviews.llvm.org/D22144 by @jlpeyton.
The patch was accepted but not landed.
This is useful functionality and I would like to use this to enable lit tests for environment variable behaviour.
Differential Revision: https://reviews.llvm.org/D36403
llvm-svn: 311180
Multi-configuration CMake generators such as those for Visual Studio or Xcode do not
specify a build config at configure time, but let the user choose at build
time. In these cases binaries go into build/${Configuration}/bin rather than
build/bin. Prior to this commit, check-lit would fail when using multi-configuration
generators as it did not know how to resolve ${Configuration} in order
to find tools such as FileCheck. This commit teaches it to resolve
llvm_tools_dir within lit using the value specified with --param
build_mode.
Differential Revision: https://reviews.llvm.org/D36263
llvm-svn: 309967
Summary:
This is an alternative solution to running the lit test suite on bots
without polluting the source directory. Each input test suite gets an
auto-generated site config in the build directory that points back to
the test input source directory.
This adds some cmake comlexity, but now we don't need to remove and
re-copy the test input directory before every test.
Reviewers: delcypher, modocache
Subscribers: mgorny, llvm-commits
Differential Revision: https://reviews.llvm.org/D36026
llvm-svn: 309602
Traceback (most recent call last):
File "llvm/utils/lit/tests/Inputs/shtest-format/external_shell/write-bad-encoding.py", line 5, in <module>
sys.stdout.write(b"a line with bad encoding: \xc2.")
sys.stdout.write doesn't accept bytes but sys.stdout.buffer.write accepts.
llvm-svn: 309473
This should fix googletest-format test failures on the clang modules
buildbots, which have a stale copy of the OneTest script in the build
directory.
llvm-svn: 309432
When using win32 cmd.exe, turn off command echoing at the beginning of
the script (@echo off).
Replace a bash shell script with a python script for the
fail_with_bad_encoding test.
llvm-svn: 309399
Summary:
The technique of directly calling subprocess.Popen on a python script
doesn't work on Windows. The executable path of the command must refer
to a valid win32 executable.
Instead, rename all the python scripts masquerading as gtest executables
to have .py extensions, so we can easily detect then and call the python
executable for them. Do this on Linux as well as Windows for
consistency.
The test suite directory names also come out in lower-case on Windows.
We can consider removing that in a later patch. This change just updates
the FileCheck lines to match on Windows.
Fixes PR33933
Reviewers: modocache, mgorny
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D35909
llvm-svn: 309347
Summary:
Normally Python converts all newline characters, Windows or Unix,
to Unix newlines when opening a file. However, lit opens files in
binary mode, which does not perform this conversion. As a result,
trailing Windows newlines are not stripped from test input, which
caused a failure in the TestRunner unit test:
```
FAIL: test_custom (__main__.TestIntegratedTestKeywordParser)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Users\bgesiak\src\llvm\llvm\utils\lit\tests\unit\TestRunner.py", line 109, in test_custom
self.assertItemsEqual(value, ['a', 'b', 'c'])
AssertionError: Element counts were not equal:
First has 1, Second has 0: 'c\r'
First has 0, Second has 1: 'c'
```
Fix the discrepancy in behavior across the two platforms by
manually stripping Windows newlines before yielding each line in
the test file.
Reviewers: echristo, beanz, ddunbar, delcypher, rnk
Reviewed By: rnk
Subscribers: mehdi_amini, llvm-commits
Differential Revision: https://reviews.llvm.org/D27746
llvm-svn: 309312
Summary:
Depends on https://reviews.llvm.org/D35879.
This reverts rL257268, which in turn was a revert of rL257221.
https://reviews.llvm.org/D35879 marks the tests in the lit test suite
that fail on Windows as XFAIL, which should allow these tests to pass
on Windows-based buildbots.
Reviewers: delcypher, beanz, mgorny, jroelofs, rnk
Reviewed By: mgorny
Subscribers: rnk, ddunbar, george.karpenkov, llvm-commits
Differential Revision: https://reviews.llvm.org/D35880
llvm-svn: 309310
Summary:
An expectation in `utils/lit/tests/Inputs/shtest-shell/redirects.txt`
expects that first a string printed to stdout is seen, and then a
string printed to stderr. Add `flush()` calls to ensure that stdout is
printed before stderr, as expected.
Reviewers: rnk, mgorny, jroelofs
Reviewed By: rnk
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D35947
llvm-svn: 309292
Reid Kleckner