ENH - Automatic support of L2 regulrization in Penalties #150
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Given a penalty$f: \mathbb{R}^n \rightarrow \mathbb{R}$ , that is already implemented in the package,$\Omega = f + \frac{\mu}{2} \lVert \cdot \rVert$
It is possible to endow it with L2 regularization to get
Indeed for a step,$\sigma$ and gradient $\mathrm{grad}$ ,$f$
the proximal operator and distance to subdifferential can be written using
prox
andsubdiffdistance
of$$
\mathrm{prox}{\Omega, \sigma}(x) = \mathrm{prox}{f, \frac{\sigma}{1 + \sigma \mu}}(\frac{x}{1 + \sigma \mu})
$$
$$
\mathrm{dist}{\partial \Omega(x)}(-\mathrm{grad}) = \mathrm{dist}{\partial f(x)}(-\mathrm{grad} - \mu x)
$$
Implementation
This can be implemented either through inheritance or a class decorator.
This PR provides a POC of the second approach. Hence to add support for L2 regularization, one only needs to decorate the penalty with
overload_with_l2
.Help needed
I unittested to the logic and implementation and everything works as expected. However, I'm running into problems when jit-compiling the class as numba doesn't support
*args
,**kwargs
, which are mandatory to overload the constructor of the penalty.Any workaround to bypass that?