Changes

doc/quickstart.md

@@ -1,4 +1,4 @@
-# A Beginner's Guide to Proper Benchmarking with BenchExec
+# A Beginner's Guide to Reliable Benchmarking
 
 > If your current setup looks similar to the below example (or you are thinking
 > about such a setup), we strongly recommend following this guide for a much
@@ -16,15 +16,23 @@ these tools. If you want to benchmark large number of runs or get the most out
 of what BenchExec provides as a benchmarking framework, consider using the tool
 `benchexec` instead (further details below).
 
-As an example, suppose that you want to measure the performance of your
-tool `program` with arguments `--foo` and `--bar` on the input files
-`input_1.in` to `input_9.in`. To measure the runtime of the tool, one may run
-`$ /usr/bin/time program --foo input_1.in` etc. and note the results. In case
-resource limitations are desired (e.g. limiting to 1 CPU and 60 sec of
-wallclock time), the calls might be
+As a concrete example, suppose that you want to measure the performance of your
+newly implemented tool `program` with arguments `--foo` and `--bar` on the
+input files `input_1.in` to `input_9.in`. To measure the runtime of the tool,
+you might run `$ /usr/bin/time program --foo input_1.in` etc. and note the
+results. In case resource limitations are desired (e.g. limiting to 1 CPU and
+60 sec of wallclock time), the calls might be
 `$ taskset -c 0 timeout 60s /usr/bin/time program ...` or similar.
 
-## Benchmarking with BenchExec
+While useful, these utilities are not suitable for reliable benchmarking,
+especially when parallelism or sub-processes are involved. For further details,
+we recommend the
+[overview slides](https://www.sosy-lab.org/research/prs/Latest_ReliableBenchmarking.pdf)
+and [the corresponding paper](https://doi.org/10.1007/s10009-017-0469-y) for a
+more complete picture of the peculiarities and pitfalls of reliable
+benchmarking.
+
+## Reliable Benchmarking with BenchExec
 
 The following steps guide you to increase the reliability and quality of
 measurements drastically by using BenchExec instead of these small
@@ -54,14 +62,8 @@ think about which executions you want to measure, what resource limits should
 be placed on the benchmarked tool(s), such as CPU time, CPU count, memory, etc.
 Also consider how timeouts should be treated.
 
-Independently of using BenchExec, we strongly recommend the
-[benchmarking guide](benchmarking.md) as well as the
-[overview slides](https://www.sosy-lab.org/research/prs/Latest_ReliableBenchmarking.pdf)
-and [the corresponding paper](https://doi.org/10.1007/s10009-017-0469-y) for a
-more complete picture of the peculiarities and pitfalls of reliable
-benchmarking. Especially for large setups, we recommend using the tool
-`benchexec`, which mitigates a lot of issues of multiple and parallel
-invocations.
+Independently of using BenchExec, we strongly recommend the following the
+guidelines of the [benchmarking guide](benchmarking.md).
 
 ### Step 3. Gather Measurements using runexec