-
Notifications
You must be signed in to change notification settings - Fork 0
/
TestingLibcxx.html
342 lines (311 loc) · 20.3 KB
/
TestingLibcxx.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Testing libc++ — libc++ 11.0 documentation</title>
<link rel="stylesheet" href="_static/haiku.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<script type="text/javascript" src="_static/language_data.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Feature Test Macro Support" href="FeatureTestMacroTable.html" />
<link rel="prev" title="Building libc++" href="BuildingLibcxx.html" />
</head><body>
<div class="header" role="banner"><h1 class="heading"><a href="index.html">
<span>libc++ 11.0 documentation</span></a></h1>
<h2 class="heading"><span>Testing libc++</span></h2>
</div>
<div class="topnav" role="navigation" aria-label="top navigation">
<p>
«  <a href="BuildingLibcxx.html">Building libc++</a>
  ::  
<a class="uplink" href="index.html">Contents</a>
  ::  
<a href="FeatureTestMacroTable.html">Feature Test Macro Support</a>  »
</p>
</div>
<div class="content">
<div class="section" id="testing-libc">
<h1>Testing libc++<a class="headerlink" href="#testing-libc" title="Permalink to this headline">¶</a></h1>
<div class="contents local topic" id="contents">
<ul class="simple">
<li><a class="reference internal" href="#getting-started" id="id1">Getting Started</a><ul>
<li><a class="reference internal" href="#setting-up-the-environment" id="id2">Setting up the Environment</a></li>
<li><a class="reference internal" href="#example-usage" id="id3">Example Usage</a></li>
</ul>
</li>
<li><a class="reference internal" href="#lit-options" id="id4">LIT Options</a><ul>
<li><a class="reference internal" href="#command-line-options" id="id5">Command Line Options</a></li>
<li><a class="reference internal" href="#environment-variables" id="id6">Environment Variables</a></li>
</ul>
</li>
<li><a class="reference internal" href="#benchmarks" id="id7">Benchmarks</a><ul>
<li><a class="reference internal" href="#building-benchmarks" id="id8">Building Benchmarks</a></li>
<li><a class="reference internal" href="#running-benchmarks" id="id9">Running Benchmarks</a></li>
</ul>
</li>
</ul>
</div>
<div class="section" id="getting-started">
<h2><a class="toc-backref" href="#id1">Getting Started</a><a class="headerlink" href="#getting-started" title="Permalink to this headline">¶</a></h2>
<p>libc++ uses LIT to configure and run its tests.</p>
<p>The primary way to run the libc++ tests is by using <cite>make check-libcxx</cite>.</p>
<p>However since libc++ can be used in any number of possible
configurations it is important to customize the way LIT builds and runs
the tests. This guide provides information on how to use LIT directly to
test libc++.</p>
<p>Please see the <a class="reference external" href="http://llvm.org/docs/CommandGuide/lit.html">Lit Command Guide</a> for more information about LIT.</p>
<div class="section" id="setting-up-the-environment">
<h3><a class="toc-backref" href="#id2">Setting up the Environment</a><a class="headerlink" href="#setting-up-the-environment" title="Permalink to this headline">¶</a></h3>
<p>After building libc++ you must setup your environment to test libc++ using
LIT.</p>
<ol class="arabic">
<li><p class="first">Create a shortcut to the actual lit executable so that you can invoke it
easily from the command line.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ <span class="nb">alias</span> <span class="nv">lit</span><span class="o">=</span><span class="s1">'python path/to/llvm/utils/lit/lit.py'</span>
</pre></div>
</div>
</li>
<li><p class="first">Tell LIT where to find your build configuration.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ <span class="nb">export</span> <span class="nv">LIBCXX_SITE_CONFIG</span><span class="o">=</span>path/to/build-libcxx/test/lit.site.cfg
</pre></div>
</div>
</li>
</ol>
</div>
<div class="section" id="example-usage">
<h3><a class="toc-backref" href="#id3">Example Usage</a><a class="headerlink" href="#example-usage" title="Permalink to this headline">¶</a></h3>
<p>Once you have your environment set up and you have built libc++ you can run
parts of the libc++ test suite by simply running <cite>lit</cite> on a specified test or
directory. For example:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ <span class="nb">cd</span> path/to/src/libcxx
$ lit -sv test/std/re <span class="c1"># Run all of the std::regex tests</span>
$ lit -sv test/std/depr/depr.c.headers/stdlib_h.pass.cpp <span class="c1"># Run a single test</span>
$ lit -sv test/std/atomics test/std/threads <span class="c1"># Test std::thread and std::atomic</span>
</pre></div>
</div>
<p>Sometimes you’ll want to change the way LIT is running the tests. Custom options
can be specified using the <cite>–param=<name>=<val></cite> flag. The most common option
you’ll want to change is the standard dialect (ie -std=c++XX). By default the
test suite will select the newest C++ dialect supported by the compiler and use
that. However if you want to manually specify the option like so:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ lit -sv test/std/containers <span class="c1"># Run the tests with the newest -std</span>
$ lit -sv --param<span class="o">=</span><span class="nv">std</span><span class="o">=</span>c++03 test/std/containers <span class="c1"># Run the tests in C++03</span>
</pre></div>
</div>
<p>Occasionally you’ll want to add extra compile or link flags when testing.
You can do this as follows:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ lit -sv --param<span class="o">=</span><span class="nv">compile_flags</span><span class="o">=</span><span class="s1">'-Wcustom-warning'</span>
$ lit -sv --param<span class="o">=</span><span class="nv">link_flags</span><span class="o">=</span><span class="s1">'-L/custom/library/path'</span>
</pre></div>
</div>
<p>Some other common examples include:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># Specify a custom compiler.</span>
$ lit -sv --param<span class="o">=</span><span class="nv">cxx_under_test</span><span class="o">=</span>/opt/bin/g++ test/std
<span class="c1"># Enable warnings in the test suite</span>
$ lit -sv --param<span class="o">=</span><span class="nv">enable_warnings</span><span class="o">=</span><span class="nb">true</span> test/std
<span class="c1"># Use UBSAN when running the tests.</span>
$ lit -sv --param<span class="o">=</span><span class="nv">use_sanitizer</span><span class="o">=</span>Undefined
</pre></div>
</div>
</div>
</div>
<div class="section" id="lit-options">
<h2><a class="toc-backref" href="#id4">LIT Options</a><a class="headerlink" href="#lit-options" title="Permalink to this headline">¶</a></h2>
<p><strong class="program">lit</strong> [<em>options</em>…] [<em>filenames</em>…]</p>
<div class="section" id="command-line-options">
<h3><a class="toc-backref" href="#id5">Command Line Options</a><a class="headerlink" href="#command-line-options" title="Permalink to this headline">¶</a></h3>
<p>To use these options you pass them on the LIT command line as –param NAME or
–param NAME=VALUE. Some options have default values specified during CMake’s
configuration. Passing the option on the command line will override the default.</p>
<dl class="option">
<dt id="cmdoption-lit-arg-cxx-under-test">
<code class="descname">cxx_under_test</code><code class="descclassname">=<path/to/compiler></code><a class="headerlink" href="#cmdoption-lit-arg-cxx-under-test" title="Permalink to this definition">¶</a></dt>
<dd><p>Specify the compiler used to build the tests.</p>
</dd></dl>
<dl class="option">
<dt id="cmdoption-lit-arg-cxx-stdlib-under-test">
<code class="descname">cxx_stdlib_under_test</code><code class="descclassname">=<stdlib name></code><a class="headerlink" href="#cmdoption-lit-arg-cxx-stdlib-under-test" title="Permalink to this definition">¶</a></dt>
<dd><p><strong>Values</strong>: libc++, libstdc++</p>
<p>Specify the C++ standard library being tested. Unless otherwise specified
libc++ is used. This option is intended to allow running the libc++ test
suite against other standard library implementations.</p>
</dd></dl>
<dl class="option">
<dt id="cmdoption-lit-arg-std">
<code class="descname">std</code><code class="descclassname">=<standard version></code><a class="headerlink" href="#cmdoption-lit-arg-std" title="Permalink to this definition">¶</a></dt>
<dd><p><strong>Values</strong>: c++98, c++03, c++11, c++14, c++17, c++2a</p>
<p>Change the standard version used when building the tests.</p>
</dd></dl>
<dl class="option">
<dt id="cmdoption-lit-arg-libcxx-site-config">
<code class="descname">libcxx_site_config</code><code class="descclassname">=<path/to/lit.site.cfg></code><a class="headerlink" href="#cmdoption-lit-arg-libcxx-site-config" title="Permalink to this definition">¶</a></dt>
<dd><p>Specify the site configuration to use when running the tests. This option
overrides the environment variable LIBCXX_SITE_CONFIG.</p>
</dd></dl>
<dl class="option">
<dt id="cmdoption-lit-arg-cxx-headers">
<code class="descname">cxx_headers</code><code class="descclassname">=<path/to/headers></code><a class="headerlink" href="#cmdoption-lit-arg-cxx-headers" title="Permalink to this definition">¶</a></dt>
<dd><p>Specify the c++ standard library headers that are tested. By default the
headers in the source tree are used.</p>
</dd></dl>
<dl class="option">
<dt id="cmdoption-lit-arg-cxx-library-root">
<code class="descname">cxx_library_root</code><code class="descclassname">=<path/to/lib/></code><a class="headerlink" href="#cmdoption-lit-arg-cxx-library-root" title="Permalink to this definition">¶</a></dt>
<dd><p>Specify the directory of the libc++ library to be tested. By default the
library folder of the build directory is used. This option cannot be used
when use_system_cxx_lib is provided.</p>
</dd></dl>
<dl class="option">
<dt id="cmdoption-lit-arg-cxx-runtime-root">
<code class="descname">cxx_runtime_root</code><code class="descclassname">=<path/to/lib/></code><a class="headerlink" href="#cmdoption-lit-arg-cxx-runtime-root" title="Permalink to this definition">¶</a></dt>
<dd><p>Specify the directory of the libc++ library to use at runtime. This directory
is not added to the linkers search path. This can be used to compile tests
against one version of libc++ and run them using another. The default value
for this option is <cite>cxx_library_root</cite>.</p>
</dd></dl>
<dl class="option">
<dt id="cmdoption-lit-arg-use-system-cxx-lib">
<code class="descname">use_system_cxx_lib</code><code class="descclassname">=<bool></code><a class="headerlink" href="#cmdoption-lit-arg-use-system-cxx-lib" title="Permalink to this definition">¶</a></dt>
<dd><p><strong>Default</strong>: False</p>
<p>Enable or disable testing against the installed version of libc++ library.
Note: This does not use the installed headers.</p>
</dd></dl>
<dl class="option">
<dt id="cmdoption-lit-arg-use-lit-shell">
<code class="descname">use_lit_shell</code><code class="descclassname">=<bool></code><a class="headerlink" href="#cmdoption-lit-arg-use-lit-shell" title="Permalink to this definition">¶</a></dt>
<dd><p>Enable or disable the use of LIT’s internal shell in ShTests. If the
environment variable LIT_USE_INTERNAL_SHELL is present then that is used as
the default value. Otherwise the default value is True on Windows and False
on every other platform.</p>
</dd></dl>
<dl class="option">
<dt id="cmdoption-lit-arg-compile-flags">
<code class="descname">compile_flags</code><code class="descclassname">="<list-of-args>"</code><a class="headerlink" href="#cmdoption-lit-arg-compile-flags" title="Permalink to this definition">¶</a></dt>
<dd><p>Specify additional compile flags as a space delimited string.
Note: This options should not be used to change the standard version used.</p>
</dd></dl>
<dl class="option">
<dt id="cmdoption-lit-arg-link-flags">
<code class="descname">link_flags</code><code class="descclassname">="<list-of-args>"</code><a class="headerlink" href="#cmdoption-lit-arg-link-flags" title="Permalink to this definition">¶</a></dt>
<dd><p>Specify additional link flags as a space delimited string.</p>
</dd></dl>
<dl class="option">
<dt id="cmdoption-lit-arg-debug-level">
<code class="descname">debug_level</code><code class="descclassname">=<level></code><a class="headerlink" href="#cmdoption-lit-arg-debug-level" title="Permalink to this definition">¶</a></dt>
<dd><p><strong>Values</strong>: 0, 1</p>
<p>Enable the use of debug mode. Level 0 enables assertions and level 1 enables
assertions and debugging of iterator misuse.</p>
</dd></dl>
<dl class="option">
<dt id="cmdoption-lit-arg-use-sanitizer">
<code class="descname">use_sanitizer</code><code class="descclassname">=<sanitizer name></code><a class="headerlink" href="#cmdoption-lit-arg-use-sanitizer" title="Permalink to this definition">¶</a></dt>
<dd><p><strong>Values</strong>: Memory, MemoryWithOrigins, Address, Undefined</p>
<p>Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when
building libc++ then that sanitizer will be used by default.</p>
</dd></dl>
<dl class="option">
<dt id="cmdoption-lit-arg-color-diagnostics">
<code class="descname">color_diagnostics</code><code class="descclassname"></code><a class="headerlink" href="#cmdoption-lit-arg-color-diagnostics" title="Permalink to this definition">¶</a></dt>
<dd><p>Enable the use of colorized compile diagnostics. If the color_diagnostics
option is specified or the environment variable LIBCXX_COLOR_DIAGNOSTICS is
present then color diagnostics will be enabled.</p>
</dd></dl>
<dl class="option">
<dt id="cmdoption-lit-arg-llvm-unwinder">
<code class="descname">llvm_unwinder</code><code class="descclassname"></code><a class="headerlink" href="#cmdoption-lit-arg-llvm-unwinder" title="Permalink to this definition">¶</a></dt>
<dd><p>Enable the use of LLVM unwinder instead of libgcc.</p>
</dd></dl>
<dl class="option">
<dt id="cmdoption-lit-arg-builtins-library">
<code class="descname">builtins_library</code><code class="descclassname"></code><a class="headerlink" href="#cmdoption-lit-arg-builtins-library" title="Permalink to this definition">¶</a></dt>
<dd><p>Path to the builtins library to use instead of libgcc.</p>
</dd></dl>
</div>
<div class="section" id="environment-variables">
<h3><a class="toc-backref" href="#id6">Environment Variables</a><a class="headerlink" href="#environment-variables" title="Permalink to this headline">¶</a></h3>
<dl class="envvar">
<dt id="envvar-LIBCXX_SITE_CONFIG=<path/to/lit.site.cfg>">
<code class="descname">LIBCXX_SITE_CONFIG=<path/to/lit.site.cfg></code><a class="headerlink" href="#envvar-LIBCXX_SITE_CONFIG=<path/to/lit.site.cfg>" title="Permalink to this definition">¶</a></dt>
<dd><p>Specify the site configuration to use when running the tests.
Also see <cite>libcxx_site_config</cite>.</p>
</dd></dl>
<dl class="envvar">
<dt id="envvar-LIBCXX_COLOR_DIAGNOSTICS">
<code class="descname">LIBCXX_COLOR_DIAGNOSTICS</code><a class="headerlink" href="#envvar-LIBCXX_COLOR_DIAGNOSTICS" title="Permalink to this definition">¶</a></dt>
<dd><p>If <code class="docutils literal notranslate"><span class="pre">LIBCXX_COLOR_DIAGNOSTICS</span></code> is defined then the test suite will attempt
to use color diagnostic outputs from the compiler.
Also see <cite>color_diagnostics</cite>.</p>
</dd></dl>
</div>
</div>
<div class="section" id="benchmarks">
<h2><a class="toc-backref" href="#id7">Benchmarks</a><a class="headerlink" href="#benchmarks" title="Permalink to this headline">¶</a></h2>
<p>Libc++ contains benchmark tests separately from the test of the test suite.
The benchmarks are written using the <a class="reference external" href="https://github.com/google/benchmark">Google Benchmark</a> library, a copy of which
is stored in the libc++ repository.</p>
<p>For more information about using the Google Benchmark library see the
<a class="reference external" href="https://github.com/google/benchmark">official documentation</a>.</p>
<div class="section" id="building-benchmarks">
<h3><a class="toc-backref" href="#id8">Building Benchmarks</a><a class="headerlink" href="#building-benchmarks" title="Permalink to this headline">¶</a></h3>
<p>The benchmark tests are not built by default. The benchmarks can be built using
the <code class="docutils literal notranslate"><span class="pre">cxx-benchmarks</span></code> target.</p>
<p>An example build would look like:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ <span class="nb">cd</span> build
$ cmake <span class="o">[</span>options<span class="o">]</span> <path to libcxx sources>
$ make cxx-benchmarks
</pre></div>
</div>
<p>This will build all of the benchmarks under <code class="docutils literal notranslate"><span class="pre"><libcxx-src>/benchmarks</span></code> to be
built against the just-built libc++. The compiled tests are output into
<code class="docutils literal notranslate"><span class="pre">build/benchmarks</span></code>.</p>
<p>The benchmarks can also be built against the platforms native standard library
using the <code class="docutils literal notranslate"><span class="pre">-DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON</span></code> CMake option. This
is useful for comparing the performance of libc++ to other standard libraries.
The compiled benchmarks are named <code class="docutils literal notranslate"><span class="pre"><test>.libcxx.out</span></code> if they test libc++ and
<code class="docutils literal notranslate"><span class="pre"><test>.native.out</span></code> otherwise.</p>
<p>Also See:</p>
<blockquote>
<div><ul class="simple">
<li><a class="reference internal" href="BuildingLibcxx.html#build-instructions"><span class="std std-ref">Building Libc++</span></a></li>
<li><a class="reference internal" href="BuildingLibcxx.html#cmake-options"><span class="std std-ref">CMake Options</span></a></li>
</ul>
</div></blockquote>
</div>
<div class="section" id="running-benchmarks">
<h3><a class="toc-backref" href="#id9">Running Benchmarks</a><a class="headerlink" href="#running-benchmarks" title="Permalink to this headline">¶</a></h3>
<p>The benchmarks must be run manually by the user. Currently there is no way
to run them as part of the build.</p>
<p>For example:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ <span class="nb">cd</span> build/benchmarks
$ make cxx-benchmarks
$ ./algorithms.libcxx.out <span class="c1"># Runs all the benchmarks</span>
$ ./algorithms.libcxx.out --benchmark_filter<span class="o">=</span>BM_Sort.* <span class="c1"># Only runs the sort benchmarks</span>
</pre></div>
</div>
<p>For more information about running benchmarks see <a class="reference external" href="https://github.com/google/benchmark">Google Benchmark</a>.</p>
</div>
</div>
</div>
</div>
<div class="bottomnav" role="navigation" aria-label="bottom navigation">
<p>
«  <a href="BuildingLibcxx.html">Building libc++</a>
  ::  
<a class="uplink" href="index.html">Contents</a>
  ::  
<a href="FeatureTestMacroTable.html">Feature Test Macro Support</a>  »
</p>
</div>
<div class="footer" role="contentinfo">
© Copyright 2011-2018, LLVM Project.
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.8.5.
</div>
</body>
</html>