Quickstart draft

README.md

@@ -8,13 +8,17 @@ SPDX-License-Identifier: Apache-2.0
 -->
 
 # BenchExec
+
 ## A Framework for Reliable Benchmarking and Resource Measurement
 
 [![Build Status](https://gitlab.com/sosy-lab/software/benchexec/badges/main/pipeline.svg)](https://gitlab.com/sosy-lab/software/benchexec/pipelines)
 [![Apache 2.0 License](https://img.shields.io/badge/license-Apache--2-brightgreen.svg)](https://www.apache.org/licenses/LICENSE-2.0)
 [![PyPI version](https://img.shields.io/pypi/v/BenchExec.svg)](https://pypi.python.org/pypi/BenchExec)
 [![DOI](https://zenodo.org/badge/30758422.svg)](https://zenodo.org/badge/latestdoi/30758422)
 
+> [!NOTE]
+> To get started with reliably benchmarking right away, follow the
+> [quickstart guide](doc/quickstart.md).
 
 **News and Updates**:
 - Two projects accepted for BenchExec as part of [Google Summer of Code](https://summerofcode.withgoogle.com/)!
@@ -43,15 +47,15 @@ In particular, BenchExec provides three major features:
 
 - execution of arbitrary commands with precise and reliable measurement
   and limitation of resource usage (e.g., CPU time and memory),
-  and isolation against other running processes  
+  and isolation against other running processes 
   (provided by [`runexec`](https://github.com/sosy-lab/benchexec/blob/main/doc/runexec.md),
   a replacement for `time` and similar tools)
 - an easy way to define benchmarks with specific tool configurations
   and resource limits,
-  and automatically executing them on large sets of input files  
+  and automatically executing them on large sets of input files 
   (provided by [`benchexec`](https://github.com/sosy-lab/benchexec/blob/main/doc/benchexec.md)
   on top of `runexec`)
-- generation of interactive tables and plots for the results  
+- generation of interactive tables and plots for the results 
   (provided by [`table-generator`](https://github.com/sosy-lab/benchexec/blob/main/doc/table-generator.md)
   for results produced with `benchexec`)
 

doc/quickstart.md

@@ -0,0 +1,98 @@
+# A Quickstart Guide to Proper Benchmarking with BenchExec
+
+This guide provides a brief summary of instructions to set up reliable
+benchmark measurements using BenchExec and important points to consider. It is
+meant for users who either want to set up benchmarking from scratch or already
+have a simple setup using, e.g., `time`, `timeout`, `taskset`, `ulimit`, etc.
+and will guide you how to use `runexec` as a simple but much more reliable
+"drop-in" replacement for these tools.
+
+## Guiding Example
+
+> [!IMPORTANT]
+> If your current setup looks similar to the below example (or you are thinking
+> about such a setup), we strongly recommend following this guide for a much
+> more reliable process.
+
+As an example, suppose that you want to measure the performance of your
+tool `program` with arguments `--foo` and `--bar` on the input files
+`input_1.in` to `input_9.in`. To measure the runtime of the tool, one may run
+```
+$ /usr/bin/time program --foo input_1.in
+$ /usr/bin/time program --bar input_1.in
+$ /usr/bin/time program --foo input_2.in
+...
+```
+etc. and note the results. In case resource limitations are desired (e.g. 
+limiting to 1 CPU and 60 sec of wallclock time), the calls might be
+```
+$ taskset -c 0 timeout 60s /usr/bin/time program ...
+```
+or similar.
+
+## Benchmarking with BenchExec
+
+The following steps guide you to increase the reliability and quality of
+measurements drastically by using BenchExec instead of these small
+utilities.
+
+### Step 1. Install BenchExec
+
+Depending on your distribution, the setup steps vary slightly.
+On Debian/Ubuntu and similar, you can
+```
+sudo add-apt-repository ppa:sosy-lab/benchmarking
+sudo apt update && sudo apt install benchexec
+```
+Otherwise, try the pip-based setup
+```
+pip3 install --user benchexec[systemd] coloredlogs
+```
+
+You should be able to run the command
+`python3 -m benchexec.check_cgroups` without issues. See
+[the installation guide](INSTALL.md) for further details and troubleshooting.
+
+### Step 2. Consider the Setup of Your Benchmark
+
+Consider and document the benchmarking process *beforehand*. In particular,
+think about which executions you want to measure, what resource limits should
+be placed on the benchmarked tool(s), such as CPU time, CPU count, memory, etc.
+Also consider how timeouts should be treated.
+
+For more complicated setups, please also refer to the
+[benchmarking setup guide](benchmarking.md). For example, in case you want to
+execute multiple benchmarks in parallel, think about how to deal with shared
+resources (e.g. the memory bus and CPU cache). For such cases, we recommend
+using `benchexec` instead, which takes care of managing parallel invocations.
+
+
+### Step 3. Gather Measurements using runexec
+
+Using the example from above, suppose that we want to limit the process to 60s
+wall time, 1 GB of memory, and cpu core 0. Then, simply run
+```
+$ runexec --quiet --walltimelimit 60s --memlimit 1GB --cores 0 --output output_1_foo.log -- \
+  program --foo input_1.in
+```
+This executes `program --foo input_1.in` and prints measurements to standard
+output, such as walltime, cputime, memory, I/O, etc. in a simple to read and
+parse format. The output of program is redirected to `output_1_foo.log`.
+
+The tool `runexec` offers several other features, run `runexec --help` for
+further information or refer to the [documentation](runexec.md).
+
+## Further Resources
+
+BenchExec provides much more beyond this core feature of reliable measurements.
+
+The tool `benchexec` provides ways to specify the complete set of benchmarks
+and allows you to run thousands of tool / input combinations with a single
+command as well as gather, aggregate, and display all the data at the same
+time. See [the documentation](benchexec.md) for further details.
+
+Additionally, `runexec` can also be accessed through a simple
+[Python API](runexec.md#integration-into-other-benchmarking-frameworks) with
+which you can integrate it programmatically in your framework.
+
+Refer to the [index](INDEX.md) for further information.