TrustRegion correctness

[YingboMa]

We should really implement all the tests in https://people.sc.fsu.edu/~jburkardt/m_src/test_nonlin/test_nonlin.html to validate our solvers.

Here's the result of the second test problem:

NewtonRaphson residual norm: 8.514332152287816e-16
TrustRegion residual norm: 0.0008969098117197063
NLsolve residual norm: 8.496216993896772e-16

The full code is:

using NonlinearSolve, NLsolve, LinearAlgebra

function powell_singular_function!(out, x, p = nothing)
    out[1] = x[1] + 10.0 * x[2]
    out[2] = sqrt(5.0) * ( x[3] - x[4] )
    out[3] = ( x[2] - 2.0 * x[3] )^2
    out[4] = sqrt(10.0) * ( x[1] - x[4] ) * ( x[1] - x[4] )
    nothing
end

n = 4
x = zeros(n)
x[1]   = -1.2
x[2:n] .= 1.0
nlprob = NonlinearProblem(powell_singular_function!, x)
out = similar(x)
powell_singular_function!(out, solve(nlprob, NewtonRaphson(), abstol=1e-15, reltol=1e-15).u, nothing)
println("NewtonRaphson residual norm: $(norm(out))")
powell_singular_function!(out, solve(nlprob, TrustRegion(), abstol=1e-15, reltol=1e-15).u, nothing)
println("TrustRegion residual norm: $(norm(out))")
sol = nlsolve(powell_singular_function!, x, xtol=1e-15, ftol=1e-15)
println("NLsolve residual norm: $(sol.residual_norm)")

[YingboMa]

Here's another example:

NewtonRaphson residual norm: 0.0
TrustRegion residual norm: 14.356339608897732
NLsolve residual norm: 4.948952112999869

function freudenstein_roth!(out, x, p = nothing)
    out[1] = x[1] - x[2]^3 + 5.0 * x[2]^2 - 2.0 * x[2] - 13.0
    out[2] = x[1] + x[2]^3 + x[2]^2 - 14.0 * x[2] - 29.0
end
x = [1.0; 0.0]
nlprob = NonlinearProblem(freudenstein_roth!, x)
out = similar(x)
freudenstein_roth!(out, solve(nlprob, NewtonRaphson(), abstol=1e-15, reltol=1e-15).u, nothing)
println("NewtonRaphson residual norm: $(norm(out))")
freudenstein_roth!(out, solve(nlprob, TrustRegion(), abstol=1e-15, reltol=1e-15).u, nothing)
println("TrustRegion residual norm: $(norm(out))")
sol = nlsolve(freudenstein_roth!, x, xtol=1e-15, ftol=1e-15)
println("NLsolve residual norm: $(sol.residual_norm)")

[Deltadahl]

Hmm, I'll look into this!

[elbert5770]

TrustRegion() does poorly on this example:

using NonlinearSolve
#using DifferentialEquations

function sys(sto, x, p)
    @show x
    x1 = x[1]
    x2 = x[2]
    p1 = p[1]
    p2 = p[2]
    sto[1] = x1/p2 - x2/p1
    sto[2] = 100.0 - p1*x1 - p2*x2
    @show sto
    return sto
end


a=0
p = [0.5,0.5]
x = [1.0,1.0]
#du = similar(p)
probN = NonlinearProblem{true}(sys,x, p)

sol = solve(probN,TrustRegion())
@show sol
sto = similar(p)
@show sys(sto,sol.u,p)

Residuals: sys(sto, sol.u, p) = [27.517280871675148, 85.87133647397435]

with NewtonRaphson(), Residuals: sys(sto, sol.u, p) = [0.0, 0.0]
with default alg (requires using DifferentialEquations): Residuals: sys(sto, sol.u, p) = [0.0, 5.179430218049674e-10]

[Deltadahl]

I've implemented the 23 tests in https://people.sc.fsu.edu/~jburkardt/m_src/test_nonlin/test_nonlin.html
Here is the result:

nr  Problem                                            n     NewtonRaphson       TrustRegion         LevenbergMarquardt  nlsolve
1   Generalized Rosenbrock function                    10    3.9833e-311         4.9193              4.9193              0.0
2   Powell singular function                           4     7.0435e-16          7.941               0.021633            7.0217e-16
3   Powell badly scaled function                       2     0.0                 26897.0             2.1983e-14          0.0
4   Wood function                                      4     4.1832e-14          19081.0             8550.5              8.2157e-15
5   Helical valley function                            3     6.2775e-30          50.0                50.0                4.3667e-26
6   Watson function                                    2     18860.0             3004.4              129.15              114.89
7   Chebyquad function                                 2     5.5511e-17          0.075044            0.34127             9.4369e-16
8   Brown almost linear function                       10    0.0                 0.01533             16.53               4.4409e-16
9   Discrete boundary value function                   10    5.4548e-16          0.0047026           6.5869e-15          2.7756e-16
10  Discrete integral equation function                10    2.7807e-15          1.9166e-15          9.3334e-15          5.5511e-17
11  Trigonometric function                             10    1.4753e-15          2.2808e-15          0.084117            6.8695e-16
12  Variably dimensioned function                      10    0.0                 2.8636e105          1.376e-12           0.0
13  Broyden tridiagonal function                       10    1.2803e-15          2.7487e-15          1.1633e-13          6.6613e-16
14  Broyden banded function                            10    1.4002e-15          2.1729e-15          1.6865e-13          5.5511e-16
15  Hammarling 2 by 2 matrix square root problem       4     2.2204e-16          0.32355             0.91487             2.2204e-16
16  Hammarling 3 by 3 matrix square root problem       9     2.2204e-16          0.30203             1.0506              2.2204e-16
17  Dennis and Schnabel 2 by 2 example                 2     0.0                 4.4408e-16          17.262              0.0
18  Sample problem 18                                  2     0.0                 6.2862e-9           2.1951              8.291e-17
19  Sample problem 19                                  2     9.5092e-16          3.4505e-7           76.367              7.7402e-16
20  Scalar problem f(x) = x(x - 5)^2                   1     0.0                 1.8919e-16          2.5564e-18          0.0
21  Freudenstein-Roth function                         2     0.0                 20.012              20.012              4.949
22  Boggs function                                     2     0.0                 1.6031              2.0                 3.9431e-17
23  Chandrasekhar function                             10    6.6613e-16          9.9301e-16          1.0906e-13          2.2204e-16

[ChrisRackauckas]

Handled in #210