From 9e89fd8ba02f5c61e19d3abe459bb650612219dd Mon Sep 17 00:00:00 2001 From: Tobias Meggendorfer Date: Sun, 26 May 2024 23:09:30 +0200 Subject: [PATCH] Quickstart draft --- README.md | 10 +++-- doc/quickstart.md | 98 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 105 insertions(+), 3 deletions(-) create mode 100644 doc/quickstart.md diff --git a/README.md b/README.md index 41e522caa..94660b068 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,7 @@ SPDX-License-Identifier: Apache-2.0 --> # BenchExec + ## A Framework for Reliable Benchmarking and Resource Measurement [![Build Status](https://gitlab.com/sosy-lab/software/benchexec/badges/main/pipeline.svg)](https://gitlab.com/sosy-lab/software/benchexec/pipelines) @@ -15,6 +16,9 @@ SPDX-License-Identifier: Apache-2.0 [![PyPI version](https://img.shields.io/pypi/v/BenchExec.svg)](https://pypi.python.org/pypi/BenchExec) [![DOI](https://zenodo.org/badge/30758422.svg)](https://zenodo.org/badge/latestdoi/30758422) +> [!NOTE] +> To get started with reliably benchmarking right away, follow the +> [quickstart guide](doc/quickstart.md). **News and Updates**: - Two projects accepted for BenchExec as part of [Google Summer of Code](https://summerofcode.withgoogle.com/)! @@ -43,15 +47,15 @@ In particular, BenchExec provides three major features: - execution of arbitrary commands with precise and reliable measurement and limitation of resource usage (e.g., CPU time and memory), - and isolation against other running processes + and isolation against other running processes (provided by [`runexec`](https://github.com/sosy-lab/benchexec/blob/main/doc/runexec.md), a replacement for `time` and similar tools) - an easy way to define benchmarks with specific tool configurations and resource limits, - and automatically executing them on large sets of input files + and automatically executing them on large sets of input files (provided by [`benchexec`](https://github.com/sosy-lab/benchexec/blob/main/doc/benchexec.md) on top of `runexec`) -- generation of interactive tables and plots for the results +- generation of interactive tables and plots for the results (provided by [`table-generator`](https://github.com/sosy-lab/benchexec/blob/main/doc/table-generator.md) for results produced with `benchexec`) diff --git a/doc/quickstart.md b/doc/quickstart.md new file mode 100644 index 000000000..a29ba822e --- /dev/null +++ b/doc/quickstart.md @@ -0,0 +1,98 @@ +# A Quickstart Guide to Proper Benchmarking with BenchExec + +This guide provides a brief summary of instructions to set up reliable +benchmark measurements using BenchExec and important points to consider. It is +meant for users who either want to set up benchmarking from scratch or already +have a simple setup using, e.g., `time`, `timeout`, `taskset`, `ulimit`, etc. +and will guide you how to use `runexec` as a simple but much more reliable +"drop-in" replacement for these tools. + +## Guiding Example + +> [!IMPORTANT] +> If your current setup looks similar to the below example (or you are thinking +> about such a setup), we strongly recommend following this guide for a much +> more reliable process. + +As an example, suppose that you want to measure the performance of your +tool `program` with arguments `--foo` and `--bar` on the input files +`input_1.in` to `input_9.in`. To measure the runtime of the tool, one may run +``` +$ /usr/bin/time program --foo input_1.in +$ /usr/bin/time program --bar input_1.in +$ /usr/bin/time program --foo input_2.in +... +``` +etc. and note the results. In case resource limitations are desired (e.g. +limiting to 1 CPU and 60 sec of wallclock time), the calls might be +``` +$ taskset -c 0 timeout 60s /usr/bin/time program ... +``` +or similar. + +## Benchmarking with BenchExec + +The following steps guide you to increase the reliability and quality of +measurements drastically by using BenchExec instead of these small +utilities. + +### Step 1. Install BenchExec + +Depending on your distribution, the setup steps vary slightly. +On Debian/Ubuntu and similar, you can +``` +sudo add-apt-repository ppa:sosy-lab/benchmarking +sudo apt update && sudo apt install benchexec +``` +Otherwise, try the pip-based setup +``` +pip3 install --user benchexec[systemd] coloredlogs +``` + +You should be able to run the command +`python3 -m benchexec.check_cgroups` without issues. See +[the installation guide](INSTALL.md) for further details and troubleshooting. + +### Step 2. Consider the Setup of Your Benchmark + +Consider and document the benchmarking process *beforehand*. In particular, +think about which executions you want to measure, what resource limits should +be placed on the benchmarked tool(s), such as CPU time, CPU count, memory, etc. +Also consider how timeouts should be treated. + +For more complicated setups, please also refer to the +[benchmarking setup guide](benchmarking.md). For example, in case you want to +execute multiple benchmarks in parallel, think about how to deal with shared +resources (e.g. the memory bus and CPU cache). For such cases, we recommend +using `benchexec` instead, which takes care of managing parallel invocations. + + +### Step 3. Gather Measurements using runexec + +Using the example from above, suppose that we want to limit the process to 60s +wall time, 1 GB of memory, and cpu core 0. Then, simply run +``` +$ runexec --quiet --walltimelimit 60s --memlimit 1GB --cores 0 --output output_1_foo.log -- \ + program --foo input_1.in +``` +This executes `program --foo input_1.in` and prints measurements to standard +output, such as walltime, cputime, memory, I/O, etc. in a simple to read and +parse format. The output of program is redirected to `output_1_foo.log`. + +The tool `runexec` offers several other features, run `runexec --help` for +further information or refer to the [documentation](runexec.md). + +## Further Resources + +BenchExec provides much more beyond this core feature of reliable measurements. + +The tool `benchexec` provides ways to specify the complete set of benchmarks +and allows you to run thousands of tool / input combinations with a single +command as well as gather, aggregate, and display all the data at the same +time. See [the documentation](benchexec.md) for further details. + +Additionally, `runexec` can also be accessed through a simple +[Python API](runexec.md#integration-into-other-benchmarking-frameworks) with +which you can integrate it programmatically in your framework. + +Refer to the [index](INDEX.md) for further information.