FAQ.md

Frequently asked questions

1. Which R VMs does rbenchmark driver support?
2. How to run a benchmark with rbenchmark driver?
3. Can I change the default RVM used by the rbenchmark driver?
4. Why do I get incorrect result?
5. How many times does the rbenchmark driver execute the run() function?
6. Can I execute the run() function once?
7. Can I add another RVM support in rbenchmark driver?

Answers

1. Which R VMs does rbenchmark driver support?

R benchmark driver currently supports the following RVMs.

• GNU R: all versions of GNU R. Can select using byte-code compiler or not
• pqR: a pretty quick version of R. http://www.pqr-project.org/
• TERR: TIBCO Enterprise Runtime for R (TERR). Web
• Purdue FastR: https://github.com/allr/fastr
• Orcale FastR(TruffleR): https://bitbucket.org/allr/fastr
• Renjin: http://www.renjin.org/
• Riposte: https://github.com/jtalbot/riposte
• Rapydo: https://bitbucket.org/cfbolz/rapydo/src
• ORBIT: https://github.com/wanghc78/ORBIT/wiki

You can use rbench.py -h to list the supported RVMs.

2. How to run a benchmark with rbenchmark driver?

The simple command is rbench.py yourBenchmarkApp.R. In this case the driver will use the default RVMs, and execute the run() function in yourBenchmarkApp.R with pre-defined iterations.

You can change the RVM used by using --rvm argument. For all the detail usage, please refer Running Benchmark.

3. Can I change the default RVM used by the rbenchmark driver?

Yes. The supported RVMs are defined in rbench.cfg. The first RVM listed in the configuration file (the one below the [DEFAULT] section) is the default one. You can move any RVM section to the first one as the default one.

4. Why do I get incorrect result?

The most common problem is the command generated by rbench.py to invoke the RVM to execute the target benchmark R file is incorrect. For example, you may get "No such file or directory". Please check the corresponding RVM section in rbench.cfg to see whether the RVM's path is set correctly.

5. How many times does the rbenchmark driver execute the run() function?

By default, the driver will invoke your app R files two rounds. The first round (warmup round) executes run() function 2 times, and the second round (bench round) execute the run() function 7 times. Then it will diff the measured metrics, and divide the values by 5 to get the metrics for one time execution of run() function.

You can set the execution times for warmup round and bench round by using --warmup_rep and --bench_rep arguments. Or you can change the default values in rbench.cfg.

6. Can I execute the run() function once?

Yes. Most of the R benchmark applications in the repository has a default wrapper.

 if (!exists('harness_argc')) {
     n <- setup(commandArgs(TRUE))
     run(n)
 }

If the RVM support all these features in the default wrapper, you can execute the application by something like Rscript yourBenchmarkApp.R.

Another way is just set the --warmup_rep 0 and --bench_rep 1.

7. Can I add another RVM support in rbenchmark driver?

Yes. You can add another RVM support by adding a section into rbench.cfg. Please refer Running Benchmark for the detail.