SHAKER is a lightweight approach to detect flakiness in time-constrained tests by adding noise in the execution environment.
@inproceedings{STA:ICSME2020,
author = {Silva, Denini and Teixeira, Leopoldo and d'Amorim, Marcelo},
booktitle={2020 IEEE International Conference on Software Maintenance and Evolution (ICSME)},
title = {{Shake It! Detecting Flaky Tests Caused by Concurrency with Shaker}},
year = {2020},
pages={301-311},
doi={10.1109/ICSME46990.2020.00037}
}
This repository is organized in the following structure:
- the
datasetfolder contains the set of flaky tests we used in the paper; - the
raw_resultsfolder contains the raw data produced from the evaluation we performed; - the
evaluationfolder contains the toolset developed for runningShakeragainst the apps listed below and producing the raw data described above. - the
standalonefolder contains the instructions for runningShakerin other apps.
The minimum requirements for running the tools and scripts we developed for this work are described below. These steps were executed and tested using a fresh install of the Ubuntu 18.04 LTS environment. The approach relies on running an emulator, and some steps might take long to execute, although we paremeterize our scripts so it is not necessary to run the exact number of replications we executed for each experiment in the paper (see the evaluation folder).
The step by step instructions follow below, but these are all automated under the setup.sh script.
sudo apt-get update
sudo apt-get install -y openjdk-8-jdk stress-ng python3-pip unzip r-base
We assume a working Python3 environment. We require installing the following libraries for executing the Minimal Hitting-Set (MHS) algorithm.
pip3 install python-sat
pip3 install numpy==1.16.1
wget https://dl.google.com/android/repository/commandlinetools-linux-6609375_latest.zip
unzip commandlinetools-linux-6609375_latest.zip
rm -f commandlinetools-linux-6609375_latest.zip
mkdir cmdline-tools
mv tools/ cmdline-tools
mkdir Android
mv cmdline-tools/ Android
yes | $HOME/Android/cmdline-tools/tools/bin/sdkmanager --licenses
$HOME/Android/cmdline-tools/tools/bin/sdkmanager "platforms;android-28"
$HOME/Android/cmdline-tools/tools/bin/sdkmanager "system-images;android-28;default;x86"
$HOME/Android/cmdline-tools/tools/bin/sdkmanager "build-tools;28.0.3"
If you want to use another API version, such as 23, just change the numbers in platforms and system-images:
$HOME/Android/cmdline-tools/tools/bin/sdkmanager "platforms;android-28"
$HOME/Android/cmdline-tools/tools/bin/sdkmanager "system-images;android-28;default;x86"
echo no | $HOME/Android/cmdline-tools/tools/bin/avdmanager create avd --name EmuAPI28 --package "system-images;android-28;default;x86"
If you want to use another API version, such as 23, just change the corresponding numbers:
echo no | $HOME/Android/cmdline-tools/tools/bin/avdmanager create avd --name EmuAPI23 --package "system-images;android-23;default;x86"
To run the emulator created by the setup.sh script, just execute:
./runemulator.sh
This will run the DEFAULT_EMULATOR defined in the runemulator.sh. If you want to run a particular emulator that you have previously created, with a different name, change the variable inside the script or provide the name as the first argument:
./runemulator.sh AVD_NAME
Finally, if you want to run in headless mode (-no-window), you might execute the following script that runs the default emulator:
./runemulator-nogui.sh
Again, if you want to run a particular emulator that you have previously created, with a different name, change the variable inside the script or provide it as the first argument:
./runemulator-nogui.sh AVD_NAME
After finishing booting the emulator for the first time (wait for the boot complete message in the console), the apps we have used for the evaluation can be installed using the install-apps.sh script.
This is all it takes for configuring the environment for running Shaker by using the predefined scripts for each research question from the paper, which are available at the evaluation folder. You can also run the icsme2020-eval.sh or quick-eval.sh to go through all RQs.
For each project where we found flaky tests, we opened issues: