Optimal testing in the experiment-rich regime

Code to replicate results in "Optimal Testing in the Experiment-rich Regime" paper

Requirements

• Python 3
• pandas
• dpfly
• matplotlib
• numpy
• scipy
• toolz
• click

How-to

Generating data from simulations

• Create experiments folder
• Run all the bash scripts to generate outcomes for tests. This can take many hours
• Run the summarize_experiments.py script to parse outcomes

Note, the parsed experiments from our simulations can be found in data/experiment_summary.csv

Creating plots

• Create plots folder
• Run create_baseball_plot.py to generate the baseball result plot
• Run crease_boundary_plot.py to generate the boundary plot including the heuristic boundary