More trust region schemes

[YingboMa]

The current scheme for trust region radius selection will force the algorithm to stuck at local minima as shown in #146

New trust region radius calculations are suggested in:

https://www.tandfonline.com/doi/full/10.1080/10556788.2014.932943

https://link.springer.com/article/10.1007/s10107-015-0893-2#Sec4

[yash2798]

Hi, @YingboMa , I am working on this issue. In Yuan's paper there are three different strategies laid out to update the trust region radius (2.10, 2.12 and 2.14). Which one should be used here? I ask this because these methods are locally superlinearly convergent but nothing is said about global convergence in the paper.

[ChrisRackauckas]

Implement them as different options. I wouldn't trust their benchmarking too much and we should benchmark it ourselves in real contexts

[yash2798]

Okay! and by different options, do you mean implementing the trust_region_step! function as multiple dispatches?

[ChrisRackauckas]

Yes, so the TrustRegion(tralg = TRYuan()) or something, so that the same algorithm can swap to different trust_region_step!s. Then we can try them on a bunch of real problems and see if it aligns with what the papers say will happen, which happens about 1/4 of the time and 3/4 of the time we learn something that isn't in the paper 😅

[ChrisRackauckas]

Handled in #210