Commit Graph

75 Commits

Author SHA1 Message Date
Todd Fiala 2d3754d82e test runner: switch to pure-Python timeout mechanism
The timeout mechanism has been implemented in python,
so there is no more dependence on external utilities
for timing out an inferior dotest.py process.

Platform-specific bits are implemented for POSIX-like
and Windows platforms.  There is a small class named
ProcessHelper in process_control.py that needs to be
overridden for platforms that fall outside of the
POSIX-like and Windows platforms.

See http://reviews.llvm.org/D13124 for more details.

llvm-svn: 248834
2015-09-29 22:19:06 +00:00
Pavel Labath 8ff61200f4 Remove XTIMEOUT from TestProcessAttach on linux
llvm-svn: 248711
2015-09-28 13:27:48 +00:00
Ying Chen d93aa1060a Check existence of SIGHUP before using it
Summary: -SIGHUP doesn't exist on Windows

Reviewers: tfiala

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D13114

llvm-svn: 248434
2015-09-23 21:53:18 +00:00
Todd Fiala ea73624e5f Cleaned up results formatter options hand-off.
* --results-formatter-options renamed to --results-formatter-option,
  with short version of -O

* Multiple --results-formatter-option=OPTION can be specified.  The
  comma-separating mechanism has been removed.

* XunitFormatter options modified: -n and -r are now short forms of
  --ignore-skip-name and --ignore-skip-reason.  Those long option
  names were tweaked lightly.  They also can be specified multiple
  times on the command line.  The comma-separating, multiple-pattern-
  per-option mechanism has been removed.

One can now specify:

  dotest.py --results-file stdout -O-ndsym -O-nlldb-mi

for example, to ignore reporting skips for dsym-related or lldb-mi-related
tests in the xUnit report.

llvm-svn: 248384
2015-09-23 15:21:28 +00:00
Todd Fiala 871b2e5e30 test runner: remove print from prior commit; fixup listner socket backlog
When doing test event collation from dotest inferiors to the parallel test
runner, I had a hard-coded small value for the socket backlog. Fix it to
be 2*num workers so we don't fail socket connections from inferiors.

llvm-svn: 248337
2015-09-22 22:47:34 +00:00
Todd Fiala 83c32e3a49 test framework: default to threading-based test runners
Windows gets threading-pool, OS X versions < 10.10 get
multiprocessing, everyone else gets threading.

llvm-svn: 248323
2015-09-22 21:19:40 +00:00
Todd Fiala da817b6186 Test runner process group isolation take 2.
This one does not create a session, but just creates a
new process group in the same session.

llvm-svn: 248306
2015-09-22 18:05:11 +00:00
Todd Fiala f42df67cad Roll back r248282.
I'm seeing timed out tests not properly timing out on the Linux
build bot after this change.  Not sure if it is related but
seems suspect.

llvm-svn: 248284
2015-09-22 16:03:43 +00:00
Todd Fiala 1bf2bb713b test runner: Unix systems now put inferior dotest in its own process group.
This increases isolation as it relates to signal handling between parent and
children.

llvm-svn: 248282
2015-09-22 15:21:50 +00:00
Todd Fiala de9a44e3e9 test framework: parallel test runner sends terminate to formatter before printing to stdout
The parallel test runner now sends the terminate event to the formatter
(if there is one) after the parallel test runs but before dumping anything
to stdout/stderr at the end of the run.  This allows the existing
stdout/stderr summary reporting to co-exist nicely with a formatter like
the test_results.Curses that otherwise clobbers the screen.

llvm-svn: 248228
2015-09-22 00:15:50 +00:00
Greg Clayton b0d148e9f3 Fixed up the curses test results formatter to:
- rename "Failures" window to "Completed Tests"
- Remove the extra lock that I incorrectly added to the ResultsFormatter as it already had one
- Init the curses GUI with the right number of jobs when handling the "intialize" event
- tear down the curses GUI when tests complete

llvm-svn: 248179
2015-09-21 17:25:01 +00:00
Todd Fiala 1cc97b4399 test framework: ignore sighups
On OS X, we're occasionally seeing sighups come in to what
looks like the whole test runner process group (all the
multiprocessing workers and the main process).  It's not due
to a lost console.  This change has the main parallel test runner
process and the child worker processes ignore sighup.

Covers:
https://llvm.org/bugs/show_bug.cgi?id=24846

llvm-svn: 248141
2015-09-21 05:42:26 +00:00
Chaoren Lin f3057583e8 Fix Linux bot.
llvm-svn: 248075
2015-09-19 00:57:11 +00:00
Greg Clayton 1827fc25ab Added a curses based way to see the test suite running. Works only where curses is implemented. Try it out with:
./dotest.py --results-formatter=test_results.Curses --results-file=/dev/stdout

llvm-svn: 248072
2015-09-19 00:39:09 +00:00
Todd Fiala 40b180e7ee test events: added optional value type to extra event key/val pairs
The test events had worker indexes coming across as strings.  I
want them to be ints.  worker_index now comes across as an int in
the dicationary.

The optional type can be specified with:
--event-add-entries key=val[:type][,key2=val2[:type2]...]
The type piece may be 'int' at this time.  That is all.  Otherwise
it will be a string.

llvm-svn: 248066
2015-09-18 23:46:30 +00:00
Todd Fiala e83f140833 test events: announce worker count in new initialize event
See the following for details:
http://reviews.llvm.org/D12987

llvm-svn: 248059
2015-09-18 22:45:31 +00:00
Todd Fiala 33896a9872 Adds parallel work queue index to test events, stdout/stderr results support.
See http://reviews.llvm.org/D12983 for details.

llvm-svn: 248036
2015-09-18 21:01:13 +00:00
Pavel Labath 415b605249 Remove XTIMEOUT from TestAttachDenied on linux
llvm-svn: 247877
2015-09-17 09:24:44 +00:00
Todd Fiala 68615ce606 Add JUnit/XUnit-formatted output to the lldb test run system
Also introduces the test event system into our test runner framework.
See the following for details:
http://reviews.llvm.org/D12831

llvm-svn: 247722
2015-09-15 21:38:04 +00:00
Pavel Labath 7cc26d1765 Remove XTIMEOUT from TestChangeProcessGroup on linux
llvm-svn: 247552
2015-09-14 12:31:47 +00:00
Todd Fiala 8cbeed3248 dotest.py ctrl-c support, addition of --test-runner-name option.
See http://reviews.llvm.org/D12651 for more details.

For the parallel test runner, -v now also implies --output-on-success.

llvm-svn: 247084
2015-09-08 22:22:33 +00:00
Todd Fiala fed95660f3 Roll dosep.py parallel test runner into dotest.py command line
See the following for details:
http://reviews.llvm.org/D12587

llvm-svn: 246794
2015-09-03 18:58:44 +00:00
Pavel Labath 5055685cda Fix deadlock while attaching to inferiors
Summary:
There was a race condition in the AsyncThread, where we would end up sending a vAttach
notification to the thread before it got a chance set up its listener (this can be reproduced by
adding a sleep() at the very beginning of ProcessGDBRemote::AsyncThread()). This event would then
get lost and we LLDB would deadlock. I fix this by setting up the listener early on, in the
ProcessGDBRemote constructor.

This should improve the stability of all attach tests. For now, I am removing XTIMEOUT from
TestAttachResume, and will watch the buildbots for signs of trouble.

Reviewers: clayborg, ovyalov

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D12552

llvm-svn: 246756
2015-09-03 09:36:22 +00:00
Chaoren Lin 99f25befee XTIMEOUT TestIntegerTypesExpr for Darwin.
llvm-svn: 245532
2015-08-20 01:26:57 +00:00
Chaoren Lin b0138025ec XTIMEOUT TestChangeProcessGroup for Linux.
llvm-svn: 245477
2015-08-19 18:39:25 +00:00
Chaoren Lin febef1b017 XTIMEOUT TestEvents and TestThreadStates for Windows to Android.
llvm-svn: 245464
2015-08-19 17:22:12 +00:00
Chaoren Lin d904371271 XTIMEOUT TestExitDuringStep for Darwin.
llvm-svn: 245460
2015-08-19 17:13:02 +00:00
Zachary Turner 4cceca7c15 Update dosep to print unexpected successes at the end.
llvm-svn: 245066
2015-08-14 16:45:32 +00:00
Chaoren Lin b6325d0cad Make dosep.py PEP8 compliant. NFC.
Reviewers: zturner, clayborg

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D11846

llvm-svn: 244784
2015-08-12 18:02:54 +00:00
Chaoren Lin e80372a43a Refactor dosep to use list comprehension. NFC.
Reviewers: zturner

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D11910

llvm-svn: 244783
2015-08-12 18:02:53 +00:00
Chaoren Lin 5a59e46460 Don't print number of failures and percentage if no tests ran.
Reviewers: zturner

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D11909

llvm-svn: 244782
2015-08-12 18:02:51 +00:00
Chaoren Lin ffc63b044c Make dosep output status by overwriting the same line.
Summary: Update to D11816.

Reviewers: zturner

Subscribers: zturner, lldb-commits

Differential Revision: http://reviews.llvm.org/D11843

llvm-svn: 244781
2015-08-12 18:02:49 +00:00
Greg Clayton 8c3f9c9868 Don't crash if the file we want to touch doesn't exist.
llvm-svn: 244663
2015-08-11 21:01:32 +00:00
Zachary Turner 38e64175db Allow dosep.py to print dotest.py output on success.
Previously all test output was reported by each individual
instance of dotest.py.  After a recent patch, dosep gets dotest
outptu via a pipe, and selectively decides which output to
print.

This breaks certain scripts which rely on having full output
of each dotest instance to do various parsing and/or log-scraping.

While we make no promises about the format of dotest output, it's
easy to restore this to the old behavior for now, although it is
behind a flag.  To re-enable full output, run dosep.py with the -s
option.

Differential Revision: http://reviews.llvm.org/D11816
Reviewed By: Chaoren Lin

llvm-svn: 244469
2015-08-10 17:46:11 +00:00
Chaoren Lin dd44c26d34 Flush stderr on dosep status update for Windows.
llvm-svn: 244069
2015-08-05 17:36:54 +00:00
Chaoren Lin 0b8bb3da08 Xtimeout TestProcessAttach for Linux.
llvm-svn: 242932
2015-07-22 20:52:17 +00:00
Ying Chen 93190c4c0b Enable timeout on Windows
Summary: - launch dotest with gtimeout if found on Windows

Reviewers: chaoren

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D11368

llvm-svn: 242699
2015-07-20 20:04:22 +00:00
Pavel Labath 05ab2374be Fix dosep.py on windows after r240946
Summary:
On windows, global python variables are not automatically passed to child processes. This commit
makes sure the default timeout value is available to child processes by passing it directly.
I pass the whole dotest_opts value to the children, so we can use the other members as well if we
need to do it in the future.

Reviewers: amccarth

Subscribers: lldb-commits-list

Differential Revision: http://reviews.llvm.org/D10895

llvm-svn: 241459
2015-07-06 15:57:52 +00:00
Pavel Labath fad30cf194 dosep.py: Add ability to set default test timout based on target
Summary:
Current default is 10 minutes, which causes the test suite to run very long in
case of test timeouts. On local linux, each test completes in under 90 seconds in the
slowest configuration (debug build of lldb, using debug clang to build
inferiors). I am changing the default to 4m on local targets, while retaining
the 10m timeout for remote ones.

Reviewers: sivachandra, vharron

Subscribers: tberghammer, lldb-commits

Differential Revision: http://reviews.llvm.org/D10527

llvm-svn: 240946
2015-06-29 14:16:51 +00:00
Vince Harron e884389ce8 Adding some more flakey tests to the XFAIL list
llvm-svn: 240327
2015-06-22 20:54:14 +00:00
Oleksiy Vyalov 18f4c9f5a0 Mark TestMultithreaded as XTIMEOUT on Linux.
llvm-svn: 239450
2015-06-10 01:34:25 +00:00
Chaoren Lin a4447b338c Update dosep to display dotest command invoked on failure.
llvm-svn: 239258
2015-06-07 18:50:40 +00:00
Tamas Berghammer 0b11db50dd Mark TestHelloWorld as XTIMEOUT for Linux
It is times out on the build bot ~10% of the times

llvm-svn: 238836
2015-06-02 14:45:25 +00:00
Chaoren Lin 273aea894f Dump error if dotest itself fails and no tests were run.
Reviewers: zturner, clayborg

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D10163

llvm-svn: 238771
2015-06-01 19:06:01 +00:00
Chaoren Lin 5e3ab2b95f Updated dosep.py to output progress and dump std{out,err} on test failure.
Reviewers: vharron, clayborg, zturner

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D10143

llvm-svn: 238765
2015-06-01 17:49:25 +00:00
Chaoren Lin 9d2b7f982c Fix hanging test suite when dosep.py is invoked directly.
Reviewers: zturner, sivachandra

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D10134

llvm-svn: 238594
2015-05-29 18:43:46 +00:00
Ying Chen 10ed1a9ce7 Add "Failing Tests .." line to dosep logs
Summary:
-Buildbot parser depends on this line as start flag
-Will remove the dependency from buildbot parser, but it takes some time to take effect
-Will remove this line from printout after buildbot master reconfig

Reviewers: chaoren, vharron

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D10110

llvm-svn: 238527
2015-05-28 23:51:49 +00:00
Chaoren Lin 45c17ff6c2 Remove warning when testing for timeout command.
llvm-svn: 238522
2015-05-28 23:00:10 +00:00
Zachary Turner c7a7c8acca Refactor test runner to print sub-test-case pass/fail rate.
llvm-svn: 238467
2015-05-28 19:56:26 +00:00
Ed Maste 08948137fd XTIMEOUT new TestChangeProcessGroup.py on FreeBSD
llvm-svn: 238454
2015-05-28 18:45:30 +00:00