QuCAT: Application of Combinatorial Testing to Quantum Programs

This repository contains the code to reproduce the results of the paper "Application of Combinatorial Testing to Quantum Programs" published in QRS 2021.

It also contains the original results reported in the paper.

Structure of the repository

• Folder programs contains the code of 6 original programs and their 18 faulty versions of the programs, and the visualizations of them(see Tables 2 in the paper).
• Folder inputs_500 reports the test suites generated by PICT for different programs with different strengths (where 500 is the number of running repetations in our experiment)
• Folder result/results_500 contains data of experiment in the paper (where 500 is the number of running repetations in our experiment)
  • subfolder scenario1 under the subfolders named by programs contains the test results of combinatorial testing and random testing in UsageScenario1
  • subfolder scenario2 under the subfolders named by programs contains the test results of combinatorial testing in UsageScenario2, each number represents the number of test cases needed to find a fault.
• File Scenario1_gen.py is the code for implementing combinatorial testing and random testing under UsageScenario1
• File Scenario2_gen.py is the code for implementing combinatorial testing and random testing under UsageScenario2

Installation of running the code

We use Qiskit as quantum framework and R as framework for doing statistical tests.

The following steps should guarantee to have a working installation:

• Install Anaconda. You can download Anaconda for your OS from https://www.anaconda.com/
  • e.g.
    • wget https://repo.anaconda.com/archive/Anaconda3-2020.11-Linux-x86_64.sh
    • bash Anaconda3-2020.11-Linux-x86_64.sh
• Create an environment (e.g., with name "myenv")
  • conda create -n myenv python=3.9
• Activate the environment, and install qiskit and rpy2
  • conda activate myenv
  • pip install qiskit
  • pip install rpy2
• Install PICT
  • You can download PICT from [https://github.com/Microsoft/pict]
  • You need change into your root in Scenario1_gen.py file in line 59 to generate test suites
  • You need to change into the root of model files and the root of files to store the test suites in line (details can be seen in the comments of files Scenario1_gen.txt)

Running our code

The approach can be run as follows.

First, you need to activate the conda environment:

conda activate myenv

Then, you can start the program (from the repository root) to produce test results in UsageScenario1:

python Scenario1_gen.py

and produce test results in UsageScenario2:

python Scenario2_gen.py

Results of the search will be saved in folder results

Paper

X. Wang, P. Arcaini, T. Yue and S. Ali. Application of Combinatorial Testing to Quantum Programs. In 21st International Conference on Software Quality, Reliability and Security (QRS 2021) [doi]