Set inner tolerances based on outer tolerances

[dpo]

RegularizedOptimization.jl/src/TR_alg.jl

Line 176 in 3ddb3ef

subsolver_options.ϵa = k == 1 ? 1.0e-5 : max(ϵ_subsolver, min(1e-2, sqrt(ξ1)) * ξ1)

@geoffroyleconte I think the explicit 1.0e-5 and 1.0e-2 in the line above should be based on the outer solver's absolute and relative tolerances. Why solve the subproblem more accurately than the problem itself? Since we use eps_a = 1.0e-3 in the experiments, the 1.0e-5 here may be too demanding.

[geoffroyleconte]

OK, but the 1.0e-5 is only for the first iteration. Should I introduce a parameter options.ϵ0?

[dpo]

I would just use the outer solver's ϵa.

[geoffroyleconte]

What about completely removing this line? This way the subsolver would use the $\epsilon_a$ and $\epsilon_r$ of its subsolver_options.

I don't understand the current behaviour.
We have

ϵ = options.ϵa
ϵ_subsolver = options.ϵa

but after,

ϵ += ϵr * sqrt(ξ1)
ϵ_subsolver += ϵr * sqrt(ξ1)

Maybe the ϵ increment was useful for the stopping condition sqrt(ξ1) < ϵ, but I'm not sure it's useful for ϵ_subsolver.
I did this for the subsolver tolerances to keep the existing behaviour, but I'm not sure it really makes sense.

[dpo]

Definitely feel free to experiment with subsolver tolerances. The original idea was to ask the subsolver to not work hard in the beginning, and work harder as we converge. However, it seemed important for the first sub-solve to be more or less accurate. I spent quite a bit of time experimenting with those tolerances a year or two ago, so feel free to revisit.

[geoffroyleconte]

I think I understand better now.

We have:

• options.ϵa and options.ϵr used for the stopping condition of TR. ϵ is introduced such that ϵ = options.ϵa + options.ϵr * sqrt(ξ1), and TR terminates as sqrt(ξ1) < ϵ,
• subsolver_options.ϵa and subsolver_options.ϵr that have the same use for the subsolver TRDH / R2,
• We need to modify subsolver_options.ϵa so that the first solve is more accurate (1.0e-5 here), and the other solves depend on ξ1, with a maximum value that should not be too small (ϵ_subsolver = ϵa_subsolver_init + ϵr * sqrt(ξ1) where here ϵa_subsolver_init is the initial value of subsolver_options.ϵa).

So I think we need to keep the 1.0e-5 and 1.0e-2 (or maybe add 2 additionnal tolerance in the option structure. I did not manage to get better results without these tolerances.