diff --git a/README.md b/README.md index 57bfa99a5..582405982 100644 --- a/README.md +++ b/README.md @@ -36,294 +36,8 @@ with a wide variety of benchmarked tools and hundreds of thousands benchmark run BenchExec is developed at the [Software Systems Lab](http://www.sosy-lab.org) at the [University of Passau](http://www.uni-passau.de). - ### Links +- [Documentation](https://github.com/dbeyer/benchexec/tree/master/doc/INDEX.md) - [BenchExec GitHub Repository](https://github.com/dbeyer/benchexec), use this for reporting [issues](https://github.com/dbeyer/benchexec/issues) - [BenchExec at PyPI](https://pypi.python.org/pypi/BenchExec) - - -## Documentation - -### Download and Installation -BenchExec requires at least Python 3.2. - -To install BenchExec we recommend to use the Python package installer pip -(installable for example with `sudo apt-get install pip3` on Debian/Ubuntu). - -To automatically download and install the latest stable version and its dependencies -from the [Python Packaging Index](https://pypi.python.org/pypi/BenchExec), -run this command: - - sudo pip3 install benchexec - -You can also install BenchExec only for your user with - - pip3 install --user benchexec - -In this case you probably need to add the directory where pip installs the commands -to the PATH environment by adding the following line to your `~/.profile` file: - - export PATH=~/.local/bin:$PATH - -Of course you can also install BenchExec in a virtualenv if you are familiar with Python tools. - -To install the latest development version from the -[GitHub repository](https://github.com/dbeyer/benchexec), run this command: - - pip3 install --user git+https://github.com/dbeyer/benchexec.git - - -### Setting up Cgroups -To execute benchmarks and reliably measure and limit their resource consumption, -BenchExec requires that the user which executes the benchmarks -can create and modify cgroups. - -Any Linux kernel version of the last years is -acceptable, though there have been performance improvements for the memory -controller in version 3.3, and cgroups in general are still getting improved, thus, -using a recent kernel is a good idea. - -The cgroup virtual file system is typically mounted at `/sys/fs/cgroup`. -If it is not, you can mount it with - - sudo mount -t cgroup cgroup /sys/fs/cgroup - -To give all users on the system the ability to create their own cgroups, -you can use - - sudo chmod o+wt,g+w /sys/fs/cgroup/ - -Of course permissions can also be assigned in a more fine-grained way if necessary. - -Alternatively, software such as `cgrulesengd` from -the [cgroup-bin](http://libcg.sourceforge.net/) package -can be used to setup the cgroups hierarchy. - -If your machine has swap, cgroups should be configured to also track swap memory. -If the file `memory.memsw.usage_in_bytes` does not exist in the directory -where the cgroup file system is mounted, this needs to be enabled by setting -`swapaccount=1` on the command line of the kernel. -To do so, you typically need to edit your bootloader configuration -(under Ubuntu for example in `/etc/default/grub`, line `GRUB_CMDLINE_LINUX`), -update the bootloader (`sudo update-grub`), and reboot. - -It may be that your Linux distribution already mounts the cgroup file system -and creates a cgroup hierarchy for you. -In this case you need to adjust the above commands appropriately. -To optimally use BenchExec, -the cgroup controllers `cpuacct`, `cpuset`, `freezer`, and `memory` -should be mounted and usable, -i.e., they should be listed in `/proc/self/cgroups` and the current user -should have at least the permission to create sub-cgroups of the current cgroup(s) -listed in this file for these controllers. - - -### Using runexec to Benchmark a Single Run - -BenchExec provides a program called `runexec` that can be used to -easily execute a single command while measuring its resource consumption, -similarly to the tool `time` but with more reliable time measurements -and with measurement of memory usage. -To use it, simply pass as parameters the command that should be executed -(adding `--` before the command will ensure that the arguments to the command -will not be misinterpreted as arguments to `runexec`): - - runexec -- ... - -This will start the command, write its output to the file `output.log`, -and print resource measurements to stdout. Example: - - $ runexec echo Test - 2015-03-06 12:54:01,707 - INFO - Starting command echo Test - 2015-03-06 12:54:01,708 - INFO - Writing output to output.log - exitcode=0 - returnvalue=0 - walltime=0.0024175643920898438s - cputime=0.001671s - memory=131072 - -Resource limits can be enabled with additional arguments to `runexec`, -e.g. for CPU time (`--timelimit`), wall time (`--walltimelimit`), -or memory consumption (`--memlimit`). If any of the limits is exceeded, -the started command is killed forcefully (including any child processes it started). - -`runexec` can also restrict the executed command to a set of specific CPU cores -with the parameters `--cores`, -and (on NUMA systems) to specific memory regions with `--memoryNodes`. -The IDs used for CPU cores and memory regions are the same as used by the kernel -and can be seen in the directories `/sys/devices/system/cpu` and `/sys/devices/system/node`. - -Additional parameters allow to change the name of the output file and the working directory. -The full set of available parameters can be seen with `runexec -h`. - - -### Using benchexec to Benchmark a Collection of Runs - -The program `benchexec` provides the possibility to easily benchmark -multiple executions of a tool in one go. - -#### Input for benchexec -`benchexec` uses as input an XML file that defines the command(s) to execute, -the resource limits, and the input files for which the command should be run. -A complete definition of the input format can be found in the file -[doc/benchmark.xml](doc/benchmark.xml), -and examples in [doc/benchmark-example-rand.xml](doc/benchmark-example-rand.xml) -and [doc/benchmark-example-cbmc.xml](doc/benchmark-example-cbmc.xml). -A document-type definition with a formal specification of such files can be found in -[doc/benchmark.dtd](doc/benchmark.dtd). -Such benchmark-definition files consist of a root tag `` -that has attributes for the tool to use and the resource limits. -Nested `` tags allow to specify multiple different configurations of the tool, -each of which is executed with the input files. -The input files are defined in nested `` tags, -either with `` tags (which directly specify patterns of input files) -or with `` tags (which specify text files with file-name patterns on each line). -Relative file names in these tags are interpreted as relative to the directory of the XML file. -Command-line arguments for the tool are given with `