A list of benchmark suites (and some loose kernels) used in the research related to compilers, program performance, scientific computations etc.
| Benchmark | Lang | Platform | Published in | Content | Additional info |
|---|---|---|---|---|---|
| AnghaBench | C | CPU | [DaSilva2021] | ~1 million benchmarks | Obtained from open-source repositories; for analysis of predictive compilers |
| BEEBS | C | Embedded | [Pallister2013] | 10 benchmarks | For analysis of energy consumption on embedded platforms |
| Berkeley Multimedia Workload | C | CPU (MMX/SSE/...) | [Slingerland2002] | 14 applications | Architectural support for multimedia applications |
| Embench™ | C | Embedded | [Patterson2019] | 19 benchmarks | Modern benchmarks for performance analysis built on top of selected BEEBS benchmarks |
| GridNBP 3.1 | Fortran, Java (v3.0) | CPU | [VanDerWijngaart2002] | 4 benchmarks | Evaluation of the computational grids, their communications and distributed tasks. |
| LCALS v1.0.2 | C | CPU | [Hornung2013] | 32 loops | For analysis of compiler optimisations; modern extension to Livermore loops; more information |
| Livermore loops (LFK) | FORTRAN | CPU | [McMahon1986] | 24 loops | For performance analysis |
| Livermore loops | C | CPU | 1992 | 24 loops | Port to C |
| Mälardalen WCET Benchmarks | C | CPU | [Gustafsson2010] | 35 benchmarks | For Worst-Case Execution Time (WCET) analysis |
| MediaBench | C | CPU | [Lee1997] | 19 applications | For analysis of Instruction-Level Parallelism (ILP) in compilers; source code |
| MediaBench II | C | CPU | [Fritts2005, Fritts2009] | 11 applications | General analysis of compilers on video/media related application |
| MiBench | C | Embedded | [Guthaus2001] | 6 benchmarks | For performance analysis |
| NAS Parallel Benchmarks 3.4.2 | C, Fortran, MPI, OpenMP, Java (v3.0) | CPU | [Bailey1991] | 5 kernels, 3 programs, 4 other benchmarks | Evaluation of the performance of parallel supercomputers; more information |
| PARKBENCH 2.1.1 | Fortran, MPI, PVM | CPU | [Hockney1994] | 10 measurement codes, 7 kernels, 3 application | For performance analysis of parallel architectures; more information |
| PARSEC 3.0 | C, OpenMP, pthreads | CPU | [Bienia2008] | 13 programs | For research on parallelization; other versions |
| PolyBench/ACC | C, CUDA, OpenCL, OpenACC, HMPP, OpenMP | GPU, accelerators | [Grauer-Gray2012] | 29 kernels | Updated PolyBench/GPU; GitHub |
| PolyBench/C 4.2 | C | CPU | 2010 | 30 kernels | For the analysis of performance and compiler optimisations (especially related to polyhedra compilation); more information; SourceForge |
| PolyBench/Fortran 1.0 | FORTRAN | CPU | 2011 | 30 kernels | Port to FORTRAN; more information |
| PolyBench/GPU 1.0 | C, CUDA, OpenCL, OpenHMPP | CPU, GPU | [Grauer-Gray2012] | 15 kernels | Port to heterogeneous architectures; more information |
| Rodinia v3.1 | C, OpenMP, OpenCL, CUDA | CPU, GPU | [Che2009] | 23 applications (2 under evaluations) | Performance analysis of heteregenous architectures |
| SPLASH | C | CPU | [Singh1992] | 4 kernels, 8 programs | For research on parallelization; source code unavailable |
| SPLASH-2 | C | CPU | [Woo1995] | 4 kernels, 8 programs | For research on parallelization; original source code is unavailable, several modifications: staceyson/splash2, liuyix/splash2_benchmark, andysan/splash2 |
| SPLASH-3 | C | CPU | [Sakalis2016] | 4 kernels, 8 programs | For research on parallelization |
| TSVC | FORTRAN | CPU | [Callahan1988] | 135 loops | For testing automatic vectorizing compilers; Single precision version |
| TSVC | C | CPU | [Maleki2011] | 151 loops | Extended port to C |
| TSVC 2 | C | CPU | 2015 | 151 loops | Update to TSVC in C |
| UTDSP | C | CPU | 1992 | 6 loops, 12 programs | For testing compilers on Digitial Signal Processing (DSP) applications; more information |
| VersaBench | C | Custom CPUs | [Rabbah2004] | 15 programs | For the performance analysis of new flexible CPU architectures |
- BEEBS — Bristol/Embecosm Embedded Benchmark Suite
- LCALS — Livermore Compiler Analysis Loop Suite
- LFK — Livermore FORTRAN Kernels
- NPB — NAS Parallel Benchmarks
- PARKBENCH — PARallel Kernels and BENCHmarks
- PARSEC — The Princeton Application Repository for Shared-Memory Computers
- PolyBench — The Polyhedral Benchmark Suite
- SPLASH — Stanford Parallel Applications for Shared-Memory
- TSVC — Test Suite for Vectorizing Compilers
- VersaBench — A Benchmark Suite for Versatile Architectures
- [email protected]
- Benchmarks by Nikolaos Kavvadias
- Benchmarks by lac-dcc Research Group - 260 benchmarks, 36 test suites
- [Bailey1991] D. H. Bailey et al., "The NAS parallel benchmarks summary and preliminary results," Supercomputing '91: Proceedings of the 1991 ACM/IEEE Conference on Supercomputing, 1991, pp. 158-165, doi: https://doi.org/10.1145/125826.125925.
- [Bienia2008] Christian Bienia, Sanjeev Kumar, Jaswinder Pal Singh, and Kai Li. 2008. "The PARSEC benchmark suite: characterization and architectural implications". In Proceedings of the 17th international conference on Parallel architectures and compilation techniques (PACT '08). Association for Computing Machinery, New York, NY, USA, 72–81. doi:https://doi.org/10.1145/1454115.1454128
- [Callahan1988] Callahan, D., Dongarra, J., & Levine, D. (1988). "Vectorizing compilers: a test suite and results". In Proceedings. SUPERCOMPUTING ’88 (pp. 98–105). IEEE Comput. Soc. Press. https://doi.org/10.1109/SUPERC.1988.44642
- [Che2009] S. Che et al., "Rodinia: A benchmark suite for heterogeneous computing," 2009 IEEE International Symposium on Workload Characterization (IISWC), 2009, pp. 44-54, doi: https://doi.org/10.1109/IISWC.2009.5306797
- [DaSilva2021] A. F. da Silva, B. C. Kind, J. W. de Souza Magalhães, J. N. Rocha, B. C. Ferreira Guimarães and F. M. Quinão Pereira, "ANGHABENCH: A Suite with One Million Compilable C Benchmarks for Code-Size Reduction," 2021 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), 2021, pp. 378-390, doi: https://doi.org/10.1109/CGO51591.2021.9370322
- [Fritts2005] Jason E. Fritts, Frederick W. Steiling, Joseph A. Tucek, "MediaBench II video: expediting the next generation of video systems research," Proc. SPIE 5683, Embedded Processors for Multimedia and Communications II, (8 March 2005); https://doi.org/10.1117/12.587955
- [Fritts2009] Jason E. Fritts, Frederick W. Steiling, Joseph A. Tucek, and Wayne Wolf. 2009. MediaBench II video: Expediting the next generation of video systems research. Microprocess. Microsyst. 33, 4 (June, 2009), 301–318. https://doi.org/10.1016/j.micpro.2009.02.010
- [Guthaus2001] M. R. Guthaus, J. S. Ringenberg, D. Ernst, T. M. Austin, T. Mudge and R. B. Brown, "MiBench: A free, commercially representative embedded benchmark suite," Proceedings of the Fourth Annual IEEE International Workshop on Workload Characterization. WWC-4 (Cat. No.01EX538), Austin, TX, USA, 2001, pp. 3-14, doi:https://doi.org/10.1109/WWC.2001.990739
- [Grauer-Gray2012] Scott Grauer-Gray, Lifan Xu, Robert Searles, Sudhee Ayalasomayajula, and John Cavazos. "Auto-tuning a High-Level Language Targeted to GPU Codes". Proceedings of Innovative Parallel Computing (InPar '12), 2012. https://doi.org/10.1109/InPar.2012.6339595
- [Gustafsson2010] Jan Gustafsson, Adam Betts, Andreas Ermedahl, and Björn Lisper. "The Mälardalen WCET benchmarks: Past, present and future". 10th International Workshop on Worst-Case Execution Time Analysis, WCET 2010, July 6, 2010, Brussels, Belgium. https://doi.org/10.4230/OASIcs.WCET.2010.136
- [Hockney1994] Roger Hockney, and Michael Berry. "Public International Benchmarks for Parallel Computers". Technical report. February 7, 1994. https://netlib.org/parkbench/parkbench.ps
- [Hornung2013] Richard D. Hornung and Jeffrey A. Keasler, “A Case for Improved C++ Compiler Support to Enable Performance Portability in Large Physics Simulation Codes”, LLNL-TR-635681 (2013). https://computing.llnl.gov/projects/co-design/compilersupportwhitepaper_tr-635681_cover.pdf
- [Lee1997] Chunho Lee, M. Potkonjak and W. H. Mangione-Smith, "MediaBench: a tool for evaluating and synthesizing multimedia and communications systems," Proceedings of 30th Annual International Symposium on Microarchitecture, 1997, pp. 330-335, doi:https://doi.org/10.1109/MICRO.1997.645830
- [Maleki2011] Maleki, S., Gao, Y., Garzarán, M. J., Wong, T., & Padua, D. A. (2011). "An evaluation of vectorizing compilers". Parallel Architectures and Compilation Techniques - Conference Proceedings, PACT, 7, 372–382. https://doi.org/10.1109/PACT.2011.68
- [McMahon1986] F. H. McMahon. "Livermore fortran kernels: A computer test of numerical performance range". Technical Report UCRL-53745, Lawrence Livermore National Laboratory, Livermore, CA, December 1986; COMMENT: no PDF on the web.
- [Patterson2019] David Patterson, Jeremy Bennett, Palmer Dabbelt, Cesare Garlati, G. S. Madhusudan and Trevor Mudge. "Embench™: An Evolving Benchmark Suite for Embedded IoT Computers from an Academic-Industrial Cooperative: Towards the Long Overdue and Deserved Demise of Dhrystone" (2019). https://riscv.org/wp-content/uploads/2019/06/9.25-Embench-RISC-V-Workshop-Patterson-v3.pdf
- [Pallister2013] James Pallister, Simon Hollis, Jeremy Bennett (2013). "BEEBS: Open Benchmarks for Energy Measurements on Embedded Platforms" http://arxiv.org/abs/1308.5174.
- [Rabbah2004] Rodric M. Rabbah, Ian Bratt, Krste Asanovic, and Anant Agarwal, "Versatility and VersaBench: A New Metric and a Benchmark Suite for Flexible Architectures", MIT CSAIL Technical Memo, MIT-LCS-TM-646, June 2004, http://catfish.csail.mit.edu/~rabbah/versabench/MIT-LCS-TM-646.pdf
- [Sakalis2016] C. Sakalis, C. Leonardsson, S. Kaxiras and A. Ros, "Splash-3: A properly synchronized benchmark suite for contemporary research", 2016 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2016, pp. 101-111, doi: https://www.doi.org/10.1109/ISPASS.2016.7482078.
- [Singh1992] Jaswinder Pal Singh, Wolf-Dietrich Weber, and Anoop Gupta. 1992. "SPLASH: Stanford parallel applications for shared-memory". SIGARCH Comput. Archit. News 20, 1 (March 1992), 5–44. DOI:https://doi.org/10.1145/130823.130824
- [Slingerland2002] Slingerland, N., Smith, A. Design and characterization of the Berkeley multimedia workload. Multimedia Systems 8, 315–327 (2002). https://doi.org/10.1007/s005300200052
- [VanDerWijngaart2002] Rob F. Van der Wijngaart et al. "NAS Grid Benchmarks Version 1.0". NASA Technical Report NAS-02-005. July, 2002. https://www.nas.nasa.gov/assets/pdf/techreports/2002/nas-02-005.pdf
- [Woo1995] S. C. Woo, M. Ohara, E. Torrie, J. P. Singh and A. Gupta, "The SPLASH-2 programs: characterization and methodological considerations," Proceedings 22nd Annual International Symposium on Computer Architecture, 1995, pp. 24-36, doi: https://www.doi.org/10.1109/ISCA.1995.524546.