Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Renamed the *run_config.json files #340

Closed
wants to merge 8 commits into from
34 changes: 17 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ dependencies.
Sandmark uses opam, with a static local repository, to build external
libraries and applications. It then builds any sandmark OCaml
benchmarks and any data dependencies. Following this it runs the
benchmarks as defined in the `run_config.json`
benchmarks as defined in the `sequential.json`

These stages are implemented in:

Expand All @@ -57,15 +57,15 @@ These stages are implemented in:

- Runplan: the list of benchmarks which will run along with the
measurement wrapper (e.g. orun or perf) is specified in
`run_config.json`. This config file is used to generate dune files
`sequential.json`. This config file is used to generate dune files
which will run the benchmarks.

- Build: dune is used to build all the sandmark OCaml benchmarks that
are in the `benchmarks` directory.

- Execute: dune is used to execute all the benchmarks sepcified in
the runplan using the benchmark wrapper defined in
`run_config.json` and specified via the `RUN_BENCH_TARGET` variable
`sequential.json` and specified via the `RUN_BENCH_TARGET` variable
passed to the makefile.

## Configuration of the compiler build
Expand Down Expand Up @@ -140,8 +140,8 @@ sudo setcap cap_sys_nice=ep /usr/bin/chrt

### Configuring the benchmark runs

A config file can be specified with the environment variable `RUN_CONFIG_JSON`,
and the default value is `run_config.json`. This file lists the executable to
A config file can be specified with the environment variable `SEQUENTIAL_JSON`,
and the default value is `sequential.json`. This file lists the executable to
run and the wrapper which will be used to collect data (e.g. orun or perf). You
can edit this file to change benchmark parameters or wrappers.

Expand Down Expand Up @@ -177,13 +177,13 @@ current tags are:
The benchmarking machine `turing` is an Intel Xeon Gold 5120 CPU with 64GB of
RAM housed at IITM.

The `run_config.json` file may be filtered based on the tag. For example,
The `sequential.json` file may be filtered based on the tag. For example,

```bash
$ TAG='"macro_bench"' make run_config_filtered.json
$ TAG='"macro_bench"' make sequential_filtered.json
```

filters the `run_config.json` file to only contain the benchmarks tagged as
filters the `sequential.json` file to only contain the benchmarks tagged as
`macro_bench`.

### Running benchmarks
Expand All @@ -195,15 +195,15 @@ parallel benchmarks use `multibench_parallel`. You can also setup a custom
bench and add only the benchmarks you care about.

Sandmark has support to build and execute the serial benchmarks in
byte mode. A separate `run_config_byte.json` file has been created for
byte mode. A separate `sequential_byte.json` file has been created for
the same. These benchmarks are relatively slower compared to their
native execution. You can use the following commands to run the serial
benchmarks in byte mode:

```bash
$ opam install dune.2.9.0
$ USE_SYS_DUNE_HACK=1 SANDMARK_CUSTOM_NAME=5.0.0 BUILD_BENCH_TARGET=bytebench \
RUN_CONFIG_JSON=run_config_byte.json make ocaml-versions/5.0.0+stable.bench
SEQUENTIAL_JSON=sequential_byte.json make ocaml-versions/5.0.0+stable.bench
```

We can obtain throughput and latency results for the benchmarks. For obtaining
Expand Down Expand Up @@ -261,17 +261,17 @@ You can add new benchmarks as follows:

- **Add commands to run your applications:**
Add an entry for your benchmark run to the appropriate config file;
`run_config.json` for sequential benchmarks and
`multicore_parallel_run_config.json` for parallel benchmarks.
`sequential.json` for sequential benchmarks and
`parallel_turing.json` for parallel benchmarks.

### Config files

The `*_config.json` files used to build benchmarks

- **run_config.json** : Runs sequential benchmarks with stock OCaml variants in CI and sandmark-nightly on the IITM machine(turing)
- **multicore_parallel_run_config.json** : Runs parallel benchmarks with multicore OCaml in CI and sandmark-nightly on the IITM machine(turing)
- **multicore_parallel_navajo_run_config.json** : Runs parallel benchmarks with multicore OCaml in sandmark-nightly on Navajo (AMD EPYC 7551 32-Core Processor) machine
- **micro_multicore.json** : To locally run multicore specific micro benchmarks
- **sequential.json** : Runs sequential benchmarks with stock OCaml variants in CI and sandmark-nightly on the IITM machine(turing)
- **parallel_turing.json** : Runs parallel benchmarks with multicore OCaml in CI and sandmark-nightly on the IITM machine(turing)
- **parallel_navajo.json** : Runs parallel benchmarks with multicore OCaml in sandmark-nightly on Navajo (AMD EPYC 7551 32-Core Processor) machine
- **micro.json** : To locally run multicore specific micro benchmarks

### Benchmarks status

Expand Down Expand Up @@ -360,7 +360,7 @@ work on OS X is to install GNU sed with homebrew and then update the
| PIP_DEPENDENCIES | List of Python dependencies | ```intervaltree``` | building compiler and its dependencies |
| PRE_BENCH_EXEC | Any specific commands that needed to be executed before the benchmark. For eg. `PRE_BENCH_EXEC='taskset --cpu-list 3 setarch uname -m --addr-no-randomize'` | null string | executing benchmark | RUN_BENCH_TARGET | The executable to be used to run the benchmarks | `run_orun` | executing benchmark |
| RUN_BENCH_TARGET | The executable to be used to run the benchmarks | `run_orun` | executing benchmark |
| RUN_CONFIG_JSON | Input file selection that contains the list of benchmarks | `run_config.json` | executing benchmark |
| SEQUENTIAL_JSON | Input file selection that contains the list of benchmarks | `sequential.json` | executing benchmark |
| SANDMARK_DUNE_VERSION | Default dune version to be used | 2.9.0 | building compiler and its dependencies |
| SANDMARK_OVERRIDE_PACKAGES | A list of dependency packages with versions that can be overrided (optional) | "" | building compiler and its dependencies |
| SANDMARK_REMOVE_PACKAGES | A list of dependency packages to be dynamically removed (optional) | "" | building compiler and its dependencies |
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.