Skip to content

Benchmark study on the importance of hyperparameter tuning.

Notifications You must be signed in to change notification settings

hildeweerts/hyperimp

Repository files navigation

hyperimp

Benchmark study on the importance of hyperparameter tuning: which parameters are important to tune and which can be set to a default value?

Directories

A short explanation of the directory structure of this repository.

  • experiment_1: In experiment 1, we generate random hyperparameter configurations, train simple pipelines, and upload the results to OpenML. This directory contains all files necessary to run a docker container to run an instantiation the experiment.
  • experiment_2: In experiment 2, we compare the performance of a model on a task when tuning all hyperparameters ('non-fixed') v.s. tuning all hyperparameters but leaving one hyperparameter fixed to the default value determined in experiment 1 ('fixed). This directory contains all files necessary to run a docker container to run an instantiation the experiment. One instantiation consists of a nested 10 x 5 cross validation. A random search is used to tune the hyperparameters in the inner loop.
  • batch-shipyard_1: Directions and files necessary to run experiment 1 as a Azure batch job.
  • batch-shipyard_2: Directions and files necessary to run experiment 2 as a Azure batch job.
  • hyperimp: Python module used to evaluate the experiments.
  • 01 data: Data obtained from OpenML, partly preprocessed.
  • 02 figures: All figures resulting from the analysis.

Files

  • analysis experiment 1.ipynb Analysis of the results of experiment 1. Performance data is analyzed and default values are computed.

  • analysis experiment 2.ipynb Analysis of the results of experiment 2. Includes average ranks over number of iterations, tunability, and non-inferiority tests.

  • reflection.md Some general tips, tricks, and other reflections related to performing experiments on Azure and OpenML using dockerized experiment scripts.

We thank Microsoft Azure for providing the computational resources for this study.

About

Benchmark study on the importance of hyperparameter tuning.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published