266 lines
8.0 KiB
ReStructuredText
266 lines
8.0 KiB
ReStructuredText
==============
|
|
Testing libc++
|
|
==============
|
|
|
|
.. contents::
|
|
:local:
|
|
|
|
Getting Started
|
|
===============
|
|
|
|
libc++ uses LIT to configure and run its tests. The primary way to run the
|
|
libc++ tests is by using make check-libcxx. However since libc++ can be used
|
|
in any number of possible configurations it is important to customize the way
|
|
LIT builds and runs the tests. This guide provides information on how to use
|
|
LIT directly to test libc++.
|
|
|
|
Please see the `Lit Command Guide`_ for more information about LIT.
|
|
|
|
.. _LIT Command Guide: http://llvm.org/docs/CommandGuide/lit.html
|
|
|
|
Setting up the Environment
|
|
--------------------------
|
|
|
|
After building libc++ you must setup your environment to test libc++ using
|
|
LIT.
|
|
|
|
#. Create a shortcut to the actual lit executable so that you can invoke it
|
|
easily from the command line.
|
|
|
|
.. code-block:: bash
|
|
|
|
$ alias lit='python path/to/llvm/utils/lit/lit.py'
|
|
|
|
#. Tell LIT where to find your build configuration.
|
|
|
|
.. code-block:: bash
|
|
|
|
$ export LIBCXX_SITE_CONFIG=path/to/build-libcxx/test/lit.site.cfg
|
|
|
|
Example Usage
|
|
-------------
|
|
|
|
Once you have your environment set up and you have built libc++ you can run
|
|
parts of the libc++ test suite by simply running `lit` on a specified test or
|
|
directory. For example:
|
|
|
|
.. code-block:: bash
|
|
|
|
$ cd path/to/src/libcxx
|
|
$ lit -sv test/std/re # Run all of the std::regex tests
|
|
$ lit -sv test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test
|
|
$ lit -sv test/std/atomics test/std/threads # Test std::thread and std::atomic
|
|
|
|
Sometimes you'll want to change the way LIT is running the tests. Custom options
|
|
can be specified using the `--param=<name>=<val>` flag. The most common option
|
|
you'll want to change is the standard dialect (ie -std=c++XX). By default the
|
|
test suite will select the newest C++ dialect supported by the compiler and use
|
|
that. However if you want to manually specify the option like so:
|
|
|
|
.. code-block:: bash
|
|
|
|
$ lit -sv test/std/containers # Run the tests with the newest -std
|
|
$ lit -sv --param=std=c++03 test/std/containers # Run the tests in C++03
|
|
|
|
Occasionally you'll want to add extra compile or link flags when testing.
|
|
You can do this as follows:
|
|
|
|
.. code-block:: bash
|
|
|
|
$ lit -sv --param=compile_flags='-Wcustom-warning'
|
|
$ lit -sv --param=link_flags='-L/custom/library/path'
|
|
|
|
Some other common examples include:
|
|
|
|
.. code-block:: bash
|
|
|
|
# Specify a custom compiler.
|
|
$ lit -sv --param=cxx_under_test=/opt/bin/g++ test/std
|
|
|
|
# Enable warnings in the test suite
|
|
$ lit -sv --param=enable_warnings=true test/std
|
|
|
|
# Use UBSAN when running the tests.
|
|
$ lit -sv --param=use_sanitizer=Undefined
|
|
|
|
|
|
LIT Options
|
|
===========
|
|
|
|
:program:`lit` [*options*...] [*filenames*...]
|
|
|
|
Command Line Options
|
|
--------------------
|
|
|
|
To use these options you pass them on the LIT command line as --param NAME or
|
|
--param NAME=VALUE. Some options have default values specified during CMake's
|
|
configuration. Passing the option on the command line will override the default.
|
|
|
|
.. program:: lit
|
|
|
|
.. option:: cxx_under_test=<path/to/compiler>
|
|
|
|
Specify the compiler used to build the tests.
|
|
|
|
.. option:: cxx_stdlib_under_test=<stdlib name>
|
|
|
|
**Values**: libc++, libstdc++
|
|
|
|
Specify the C++ standard library being tested. Unless otherwise specified
|
|
libc++ is used. This option is intended to allow running the libc++ test
|
|
suite against other standard library implementations.
|
|
|
|
.. option:: std=<standard version>
|
|
|
|
**Values**: c++98, c++03, c++11, c++14, c++17, c++2a
|
|
|
|
Change the standard version used when building the tests.
|
|
|
|
.. option:: libcxx_site_config=<path/to/lit.site.cfg>
|
|
|
|
Specify the site configuration to use when running the tests. This option
|
|
overrides the environment variable LIBCXX_SITE_CONFIG.
|
|
|
|
.. option:: cxx_headers=<path/to/headers>
|
|
|
|
Specify the c++ standard library headers that are tested. By default the
|
|
headers in the source tree are used.
|
|
|
|
.. option:: cxx_library_root=<path/to/lib/>
|
|
|
|
Specify the directory of the libc++ library to be tested. By default the
|
|
library folder of the build directory is used. This option cannot be used
|
|
when use_system_cxx_lib is provided.
|
|
|
|
|
|
.. option:: cxx_runtime_root=<path/to/lib/>
|
|
|
|
Specify the directory of the libc++ library to use at runtime. This directory
|
|
is not added to the linkers search path. This can be used to compile tests
|
|
against one version of libc++ and run them using another. The default value
|
|
for this option is `cxx_library_root`.
|
|
|
|
.. option:: use_system_cxx_lib=<bool>
|
|
|
|
**Default**: False
|
|
|
|
Enable or disable testing against the installed version of libc++ library.
|
|
Note: This does not use the installed headers.
|
|
|
|
.. option:: use_lit_shell=<bool>
|
|
|
|
Enable or disable the use of LIT's internal shell in ShTests. If the
|
|
environment variable LIT_USE_INTERNAL_SHELL is present then that is used as
|
|
the default value. Otherwise the default value is True on Windows and False
|
|
on every other platform.
|
|
|
|
.. option:: compile_flags="<list-of-args>"
|
|
|
|
Specify additional compile flags as a space delimited string.
|
|
Note: This options should not be used to change the standard version used.
|
|
|
|
.. option:: link_flags="<list-of-args>"
|
|
|
|
Specify additional link flags as a space delimited string.
|
|
|
|
.. option:: debug_level=<level>
|
|
|
|
**Values**: 0, 1
|
|
|
|
Enable the use of debug mode. Level 0 enables assertions and level 1 enables
|
|
assertions and debugging of iterator misuse.
|
|
|
|
.. option:: use_sanitizer=<sanitizer name>
|
|
|
|
**Values**: Memory, MemoryWithOrigins, Address, Undefined
|
|
|
|
Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when
|
|
building libc++ then that sanitizer will be used by default.
|
|
|
|
.. option:: color_diagnostics
|
|
|
|
Enable the use of colorized compile diagnostics. If the color_diagnostics
|
|
option is specified or the environment variable LIBCXX_COLOR_DIAGNOSTICS is
|
|
present then color diagnostics will be enabled.
|
|
|
|
.. option:: llvm_unwinder
|
|
|
|
Enable the use of LLVM unwinder instead of libgcc.
|
|
|
|
.. option:: builtins_library
|
|
|
|
Path to the builtins library to use instead of libgcc.
|
|
|
|
|
|
Environment Variables
|
|
---------------------
|
|
|
|
.. envvar:: LIBCXX_SITE_CONFIG=<path/to/lit.site.cfg>
|
|
|
|
Specify the site configuration to use when running the tests.
|
|
Also see `libcxx_site_config`.
|
|
|
|
.. envvar:: LIBCXX_COLOR_DIAGNOSTICS
|
|
|
|
If ``LIBCXX_COLOR_DIAGNOSTICS`` is defined then the test suite will attempt
|
|
to use color diagnostic outputs from the compiler.
|
|
Also see `color_diagnostics`.
|
|
|
|
Benchmarks
|
|
==========
|
|
|
|
Libc++ contains benchmark tests separately from the test of the test suite.
|
|
The benchmarks are written using the `Google Benchmark`_ library, a copy of which
|
|
is stored in the libc++ repository.
|
|
|
|
For more information about using the Google Benchmark library see the
|
|
`official documentation <https://github.com/google/benchmark>`_.
|
|
|
|
.. _`Google Benchmark`: https://github.com/google/benchmark
|
|
|
|
Building Benchmarks
|
|
-------------------
|
|
|
|
The benchmark tests are not built by default. The benchmarks can be built using
|
|
the ``cxx-benchmarks`` target.
|
|
|
|
An example build would look like:
|
|
|
|
.. code-block:: bash
|
|
|
|
$ cd build
|
|
$ cmake [options] <path to libcxx sources>
|
|
$ make cxx-benchmarks
|
|
|
|
This will build all of the benchmarks under ``<libcxx-src>/benchmarks`` to be
|
|
built against the just-built libc++. The compiled tests are output into
|
|
``build/benchmarks``.
|
|
|
|
The benchmarks can also be built against the platforms native standard library
|
|
using the ``-DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON`` CMake option. This
|
|
is useful for comparing the performance of libc++ to other standard libraries.
|
|
The compiled benchmarks are named ``<test>.libcxx.out`` if they test libc++ and
|
|
``<test>.native.out`` otherwise.
|
|
|
|
Also See:
|
|
|
|
* :ref:`Building Libc++ <build instructions>`
|
|
* :ref:`CMake Options`
|
|
|
|
Running Benchmarks
|
|
------------------
|
|
|
|
The benchmarks must be run manually by the user. Currently there is no way
|
|
to run them as part of the build.
|
|
|
|
For example:
|
|
|
|
.. code-block:: bash
|
|
|
|
$ cd build/benchmarks
|
|
$ make cxx-benchmarks
|
|
$ ./algorithms.libcxx.out # Runs all the benchmarks
|
|
$ ./algorithms.libcxx.out --benchmark_filter=BM_Sort.* # Only runs the sort benchmarks
|
|
|
|
For more information about running benchmarks see `Google Benchmark`_.
|