This repository contains all the code required to reproduce the results in the paper [1].
[1] Ehrhardt, M. J., & Roberts, L. (2020). Inexact Derivative-Free Optimization for Bilevel Learning. Accepted to Journal of Mathematical Imaging and Vision. Journal and arXiv preprint
The goal of [1] is to efficiently solve bilevel learning problems in image analysis. The numerical results in [1] focus on learning regularization parameters for image denoising, and learning MRI sampling patterns.
This code can also be used for hyperparameter tuning in machine learning - see [2] for details.
The most important component of this repository is a dynamic accuracy version of the derivative-free optimization solver DFO-LS from [3]. This new solver is located in src/solvers/dfols, and is based on version 1.1.1 of DFO-LS.
All code here is released under the GNU GPL, except for the Kodak image files (see kodak_dataset/README.txt for details).
As an example, below shows the result of calibrating regularization parameters for a 2D image denoising dataset (see Figure 13 of [1]).
The code requires Python 3 (our results used Python 3.6) and the following packages. The version numbers listed are those we used to generate our results.
- NumPy [1.17.0]
- SciPy [1.3.1]
- pandas [0.24.2]
- matplotlib [3.0.3]
- imageio [2.8.0]
- trustregion [1.1] - this package is optional, and improves the speed of the derivative-free solver.
There are four problems implemented here, each of which are described in [1].
- 1D image denoising - learning one regularization parameter (Section 4.2)
- 1D image denoising - learning three regularization/smoothness parameters (Section 4.2)
- 2D image denoising - learning three regularization/smoothness parameters (Section 4.3)
- 1D MRI reconstruction - learning sampling patterns (Section 4.4)
To generate all results, run the file src/run_dfols.py. This script calls the function src/dfols_wrapper.py for a specific problem (of type 1-4) defined by a file in the folder src/problem_settings, and for a specific choice of lower-level strongly convex solver (dynamic/fixed accuracy for GD or FISTA).
The results (which includes the upper-level objective decrease, total count of GD/FISTA iterations, and final reconstructions) are saved in the raw_results directory. We have saved our results for all problems except 2D denoising (due to larger file sizes) in this folder.
To plot the results in the paper, run the scripts in src/plotting (all files except plot_utils.py). To run src/plotting/denoising2d.py, you will first need to generate the 2D denoising results (calling src/dfols_wrapper.py for all files src/problem_settings/3param2d_*.json). All plots are saved in figures, in a subfolder corresponding to the problem settings file executed. This repository contains the figures as they appear in [1]. The file figures/paper_references.txt indicates the file names of each figure in [1].
[1] Ehrhardt, M. J., & Roberts, L. (2021). Inexact Derivative-Free Optimization for Bilevel Learning. Journal of Mathematical Imaging and Vision, https://doi.org/10.1007/s10851-021-01020-8. Journal and arXiv preprint
[2] Ehrhardt, M. J., & Roberts, L. (2020). Efficient Hyperparameter Tuning with Dynamic Accuracy Derivative-Free Optimization. 12th OPT Workshop on Optimization for Machine Learning at NeurIPS 2020. Workshop website and arXiv preprint
[3] Cartis, C., Fiala, J., Marteau, B., & Roberts, L., Improving the Flexibility and Robustness of Model-Based Derivative-Free Optimization Solvers, ACM Transactions on Mathematical Software, 45:3 (2019), pp. 32:1-32:41. Published paper and arXiv preprint