wangtao
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs/TestingGuide: Minimal update to describe 'lit' based regression testing 

instead of DejaGNU. Still a bit kooky, since the current test format still has
some strong Tcl roots. Oh well!

llvm-svn: 110005
This commit is contained in:
Daniel Dunbar 2010-08-02 01:20:23 +00:00
parent a9075c5ead
commit f85eb933d4
1 changed files with 84 additions and 82 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

166
llvm/docs/TestingGuide.html
View File

@ -16,22 +16,22 @@
<li><a href="#requirements">Requirements</a></li> <li><a href="#requirements">Requirements</a></li>
<li><a href="#org">LLVM testing infrastructure organization</a> <li><a href="#org">LLVM testing infrastructure organization</a>
<ul> <ul>
<li><a href="#dejagnu">DejaGNU tests</a></li> <li><a href="#regressiontests">Regression tests</a></li>
<li><a href="#testsuite">Test suite</a></li> <li><a href="#testsuite">Test suite</a></li>
</ul> </ul>
</li> </li>
<li><a href="#quick">Quick start</a> <li><a href="#quick">Quick start</a>
<ul> <ul>
<li><a href="#quickdejagnu">DejaGNU tests</a></li> <li><a href="#quickregressiontests">Regression tests</a></li>
<li><a href="#quicktestsuite">Test suite</a></li> <li><a href="#quicktestsuite">Test suite</a></li>
</ul> </ul>
</li> </li>
<li><a href="#dgstructure">DejaGNU structure</a> <li><a href="#rtstructure">Regression test structure</a>
<ul> <ul>
<li><a href="#dgcustom">Writing new DejaGNU tests</a></li> <li><a href="#rtcustom">Writing new regression tests</a></li>
<li><a href="#FileCheck">The FileCheck utility</a></li> <li><a href="#FileCheck">The FileCheck utility</a></li>
<li><a href="#dgvars">Variables and substitutions</a></li> <li><a href="#rtvars">Variables and substitutions</a></li>
<li><a href="#dgfeatures">Other features</a></li> <li><a href="#rtfeatures">Other features</a></li>
</ul> </ul>
</li> </li>
<li><a href="#testsuitestructure">Test suite structure</a></li> <li><a href="#testsuitestructure">Test suite structure</a></li>
@ -46,8 +46,7 @@
</ol> </ol>
<div class="doc_author"> <div class="doc_author">
<p>Written by John T. Criswell, <a <p>Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya Lattner</p>
href="http://llvm.x10sys.com/rspencer">Reid Spencer</a>, and Tanya Lattner</p>
</div> </div>
<!--=========================================================================--> <!--=========================================================================-->
@ -56,9 +55,9 @@
<div class="doc_text"> <div class="doc_text">
<p>This document is the reference manual for the LLVM testing infrastructure. It documents <p>This document is the reference manual for the LLVM testing infrastructure. It
the structure of the LLVM testing infrastructure, the tools needed to use it, documents the structure of the LLVM testing infrastructure, the tools needed to
and how to add and run tests.</p> use it, and how to add and run tests.</p>
</div> </div>
@ -68,17 +67,9 @@ and how to add and run tests.</p>
<div class="doc_text"> <div class="doc_text">
<p>In order to use the LLVM testing infrastructure, you will need all of the software <p>In order to use the LLVM testing infrastructure, you will need all of the
required to build LLVM, plus the following:</p> software required to build LLVM, as well
as <a href="http://python.org">Python</a> 2.4 or later.</p>
<dl>
<dt><a href="http://www.gnu.org/software/dejagnu/">DejaGNU</a></dt>
<dd>The Feature and Regressions tests are organized and run by DejaGNU.</dd>
<dt><a href="http://expect.nist.gov/">Expect</a></dt>
<dd>Expect is required by DejaGNU.</dd>
<dt><a href="http://www.tcl.tk/software/tcltk/">tcl</a></dt>
<dd>Tcl is required by DejaGNU. </dd>
</dl>
</div> </div>
@ -88,29 +79,28 @@ required to build LLVM, plus the following:</p>
<div class="doc_text"> <div class="doc_text">
<p>The LLVM testing infrastructure contains two major categories of tests: code <p>The LLVM testing infrastructure contains two major categories of tests:
fragments and whole programs. Code fragments are referred to as the "DejaGNU regression tests and whole programs. The regression tests are contained inside
tests" and are in the <tt>llvm</tt> module in subversion under the the LLVM repository itself under <tt>llvm/test</tt> and are expected to always
<tt>llvm/test</tt> directory. The whole programs tests are referred to as the pass -- they should be run before every commit. The whole programs tests are
"Test suite" and are in the <tt>test-suite</tt> module in subversion. referred to as the "LLVM test suite" and are in the <tt>test-suite</tt> module
in subversion.
</p> </p>
</div> </div>
<!-- _______________________________________________________________________ --> <!-- _______________________________________________________________________ -->
<div class="doc_subsection"><a name="dejagnu">DejaGNU tests</a></div> <div class="doc_subsection"><a name="regressiontests">Regression tests</a></div>
<!-- _______________________________________________________________________ --> <!-- _______________________________________________________________________ -->
<div class="doc_text"> <div class="doc_text">
<p>Code fragments are small pieces of code that test a specific <p>The regression tests are small pieces of code that test a specific feature of
feature of LLVM or trigger a specific bug in LLVM. They are usually LLVM or trigger a specific bug in LLVM. They are usually written in LLVM
written in LLVM assembly language, but can be written in other assembly language, but can be written in other languages if the test targets a
languages if the test targets a particular language front end (and the particular language front end (and the appropriate <tt>--with-llvmgcc</tt>
appropriate <tt>--with-llvmgcc</tt> options were used options were used at <tt>configure</tt> time of the <tt>llvm</tt> module). These
at <tt>configure</tt> time of the <tt>llvm</tt> module). These tests tests are driven by the 'lit' testing tool, which is part of LLVM.</p>
are driven by the DejaGNU testing framework, which is hidden behind a
few simple makefiles.</p>
<p>These code fragments are not complete programs. The code generated <p>These code fragments are not complete programs. The code generated
from them is never executed to determine correct behavior.</p> from them is never executed to determine correct behavior.</p>
@ -157,8 +147,8 @@ generates code.</p>
<div class="doc_text"> <div class="doc_text">
<p>The tests are located in two separate Subversion modules. The <p>The tests are located in two separate Subversion modules. The regressions
DejaGNU tests are in the main "llvm" module under the directory tests are in the main "llvm" module under the directory
<tt>llvm/test</tt> (so you get these tests for free with the main llvm tree). <tt>llvm/test</tt> (so you get these tests for free with the main llvm tree).
The more comprehensive test suite that includes whole The more comprehensive test suite that includes whole
programs in C and C++ is in the <tt>test-suite</tt> module. This module should programs in C and C++ is in the <tt>test-suite</tt> module. This module should
@ -170,10 +160,10 @@ the <tt>test-suite</tt> directory will be automatically configured.
Alternatively, you can configure the <tt>test-suite</tt> module manually.</p> Alternatively, you can configure the <tt>test-suite</tt> module manually.</p>
<!-- _______________________________________________________________________ --> <!-- _______________________________________________________________________ -->
<div class="doc_subsection"><a name="quickdejagnu">DejaGNU tests</a></div> <div class="doc_subsection"><a name="quickregressiontests">Regression tests</a></div>
<!-- _______________________________________________________________________ --> <!-- _______________________________________________________________________ -->
<p>To run all of the simple tests in LLVM using DejaGNU, use the master Makefile <p>To run all of the LLVM regression tests, use master Makefile in
in the <tt>llvm/test</tt> directory:</p> the <tt>llvm/test</tt> directory:</p>
<div class="doc_code"> <div class="doc_code">
<pre> <pre>
@ -189,26 +179,14 @@ Alternatively, you can configure the <tt>test-suite</tt> module manually.</p>
</pre> </pre>
</div> </div>
<p>To run only a subdirectory of tests in <tt>llvm/test</tt> using DejaGNU (ie. <p>If you have <a href="http://clang.llvm.org">Clang</a> checked out and built,
Transforms), just set the TESTSUITE variable to the path of the you can run the LLVM and Clang tests simultaneously using:</p>
subdirectory (relative to <tt>llvm/test</tt>):</p>
<p>or</p>
<div class="doc_code"> <div class="doc_code">
<pre> <pre>
% gmake TESTSUITE=Transforms check % gmake check-all
</pre>
</div>
<p><b>Note: If you are running the tests with <tt>objdir != subdir</tt>, you
must have run the complete testsuite before you can specify a
subdirectory.</b></p>
<p>To run only a single test, set <tt>TESTONE</tt> to its path (relative to
<tt>llvm/test</tt>) and make the <tt>check-one</tt> target:</p>
<div class="doc_code">
<pre>
% gmake TESTONE=Feature/basictest.ll check-one
</pre> </pre>
</div> </div>
@ -221,6 +199,27 @@ subdirectory.</b></p>
</pre> </pre>
</div> </div>
<p>To run individual tests or subsets of tests, you can use the 'llvm-lit'
script which is built as part of LLVM. For example, to run the
'Integer/BitCast.ll' test by itself you can run:</p>
<div class="doc_code">
<pre>
% llvm-lit ~/llvm/test/Integer/BitCast.ll
</pre>
</div>
<p>or to run all of the ARM CodeGen tests:</p>
<div class="doc_code">
<pre>
% llvm-lit ~/llvm/test/CodeGen/ARM
</pre>
</div>
<p>For more information on using the 'lit' tool, see 'llvm-lit --help' or the
'lit' man page.</p>
<!-- _______________________________________________________________________ --> <!-- _______________________________________________________________________ -->
<div class="doc_subsection"><a name="quicktestsuite">Test suite</a></div> <div class="doc_subsection"><a name="quicktestsuite">Test suite</a></div>
<!-- _______________________________________________________________________ --> <!-- _______________________________________________________________________ -->
@ -274,11 +273,11 @@ that subdirectory.</p>
</div> </div>
<!--=========================================================================--> <!--=========================================================================-->
<div class="doc_section"><a name="dgstructure">DejaGNU structure</a></div> <div class="doc_section"><a name="rtstructure">Regression test structure</a></div>
<!--=========================================================================--> <!--=========================================================================-->
<div class="doc_text"> <div class="doc_text">
<p>The LLVM DejaGNU tests are driven by DejaGNU together with GNU Make and are <p>The LLVM regression tests are driven by 'lit' and are located in
located in the <tt>llvm/test</tt> directory. the <tt>llvm/test</tt> directory.
<p>This directory contains a large array of small tests <p>This directory contains a large array of small tests
that exercise various features of LLVM and to ensure that regressions do not that exercise various features of LLVM and to ensure that regressions do not
@ -301,23 +300,24 @@ that subdirectory.</p>
</div> </div>
<!-- _______________________________________________________________________ --> <!-- _______________________________________________________________________ -->
<div class="doc_subsection"><a name="dgcustom">Writing new DejaGNU tests</a></div> <div class="doc_subsection"><a name="rtcustom">Writing new regression tests</a></div>
<!-- _______________________________________________________________________ --> <!-- _______________________________________________________________________ -->
<div class="doc_text"> <div class="doc_text">
<p>The DejaGNU structure is very simple, but does require some information to <p>The regression test structure is very simple, but does require some
be set. This information is gathered via <tt>configure</tt> and is written information to be set. This information is gathered via <tt>configure</tt> and
to a file, <tt>site.exp</tt> in <tt>llvm/test</tt>. The <tt>llvm/test</tt> is written to a file, <tt>lit.site.cfg</tt>
Makefile does this work for you.</p> in <tt>llvm/test</tt>. The <tt>llvm/test</tt> Makefile does this work for
you.</p>
<p>In order for DejaGNU to work, each directory of tests must have a <p>In order for the regression tests to work, each directory of tests must
<tt>dg.exp</tt> file. DejaGNU looks for this file to determine how to run the have a <tt>dg.exp</tt> file. Lit looks for this file to determine how to
tests. This file is just a Tcl script and it can do anything you want, but run the tests. This file is just a Tcl script and it can do anything you want,
we've standardized it for the LLVM regression tests. If you're adding a but we've standardized it for the LLVM regression tests. If you're adding a
directory of tests, just copy <tt>dg.exp</tt> from another directory to get directory of tests, just copy <tt>dg.exp</tt> from another directory to get
running. The standard <tt>dg.exp</tt> simply loads a Tcl running. The standard <tt>dg.exp</tt> simply loads a Tcl library
library (<tt>test/lib/llvm.exp</tt>) and calls the <tt>llvm_runtests</tt> (<tt>test/lib/llvm.exp</tt>) and calls the <tt>llvm_runtests</tt> function
function defined in that library with a list of file names to run. The names defined in that library with a list of file names to run. The names are
are obtained by using Tcl's glob command. Any directory that contains only obtained by using Tcl's glob command. Any directory that contains only
directories does not need the <tt>dg.exp</tt> file.</p> directories does not need the <tt>dg.exp</tt> file.</p>
<p>The <tt>llvm-runtests</tt> function lookas at each file that is passed to <p>The <tt>llvm-runtests</tt> function lookas at each file that is passed to
@ -378,7 +378,8 @@ that subdirectory.</p>
<p>There are some quoting rules that you must pay attention to when writing <p>There are some quoting rules that you must pay attention to when writing
your RUN lines. In general nothing needs to be quoted. Tcl won't strip off any your RUN lines. In general nothing needs to be quoted. Tcl won't strip off any
' or " so they will get passed to the invoked program. For example:</p> quote characters so they will get passed to the invoked program. For
example:</p>
<div class="doc_code"> <div class="doc_code">
<pre> <pre>
@ -695,7 +696,7 @@ define two separate CHECK lines that match on the same line.
</div> </div>
<!-- _______________________________________________________________________ --> <!-- _______________________________________________________________________ -->
<div class="doc_subsection"><a name="dgvars">Variables and <div class="doc_subsection"><a name="rtvars">Variables and
substitutions</a></div> substitutions</a></div>
<!-- _______________________________________________________________________ --> <!-- _______________________________________________________________________ -->
<div class="doc_text"> <div class="doc_text">
@ -791,7 +792,7 @@ substitutions</a></div>
</div> </div>
<!-- _______________________________________________________________________ --> <!-- _______________________________________________________________________ -->
<div class="doc_subsection"><a name="dgfeatures">Other Features</a></div> <div class="doc_subsection"><a name="rtfeatures">Other Features</a></div>
<!-- _______________________________________________________________________ --> <!-- _______________________________________________________________________ -->
<div class="doc_text"> <div class="doc_text">
<p>To make RUN line writing easier, there are several shell scripts located <p>To make RUN line writing easier, there are several shell scripts located
@ -817,7 +818,7 @@ substitutions</a></div>
<p>Sometimes it is necessary to mark a test case as "expected fail" or XFAIL. <p>Sometimes it is necessary to mark a test case as "expected fail" or XFAIL.
You can easily mark a test as XFAIL just by including <tt>XFAIL: </tt> on a You can easily mark a test as XFAIL just by including <tt>XFAIL: </tt> on a
line near the top of the file. This signals that the test case should succeed line near the top of the file. This signals that the test case should succeed
if the test fails. Such test cases are counted separately by DejaGnu. To if the test fails. Such test cases are counted separately by the testing tool. To
specify an expected fail, use the XFAIL keyword in the comments of the test specify an expected fail, use the XFAIL keyword in the comments of the test
program followed by a colon and one or more regular expressions (separated by program followed by a colon and one or more regular expressions (separated by
a comma). The regular expressions allow you to XFAIL the test conditionally by a comma). The regular expressions allow you to XFAIL the test conditionally by
@ -905,9 +906,10 @@ benchmarks, regression tests, code that is strange grammatically, etc. These
organizations should be relatively self explanatory.</p> organizations should be relatively self explanatory.</p>
<p>Some tests are known to fail. Some are bugs that we have not fixed yet; <p>Some tests are known to fail. Some are bugs that we have not fixed yet;
others are features that we haven't added yet (or may never add). In DejaGNU, others are features that we haven't added yet (or may never add). In the
the result for such tests will be XFAIL (eXpected FAILure). In this way, you regression tests, the result for such tests will be XFAIL (eXpected FAILure).
can tell the difference between an expected and unexpected failure.</p> In this way, you can tell the difference between an expected and unexpected
failure.</p>
<p>The tests in the test suite have no such feature at this time. If the <p>The tests in the test suite have no such feature at this time. If the
test passes, only warnings and other miscellaneous output will be generated. If test passes, only warnings and other miscellaneous output will be generated. If
@ -1143,7 +1145,7 @@ example reports that can do fancy stuff.</p>
<a href="http://validator.w3.org/check/referer"><img <a href="http://validator.w3.org/check/referer"><img
src="http://www.w3.org/Icons/valid-html401-blue" alt="Valid HTML 4.01"></a> src="http://www.w3.org/Icons/valid-html401-blue" alt="Valid HTML 4.01"></a>
John T. Criswell, Reid Spencer, and Tanya Lattner<br> John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya Lattner<br>
<a href="http://llvm.org">The LLVM Compiler Infrastructure</a><br> <a href="http://llvm.org">The LLVM Compiler Infrastructure</a><br>
Last modified: $Date$ Last modified: $Date$
</address> </address>