Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What is a Nonlinear Solver? And How to easily build Newer Ones #345

Merged
merged 76 commits into from
Jan 17, 2024

Conversation

avik-pal
Copy link
Member

@avik-pal avik-pal commented Dec 28, 2023

Goals

  • Fully Functional Iterator Interface
  • Composable Internals. Add new algorithm, the semi-implicit Continuous Newton method #330 and Simple high order multistep schemes #216 got me thinking that our level of abstraction before was way off. All our algorithms are essentially the same (okay maybe except the spectral one, but even that can be made composable and we get a bunch of those methods for free) but we keep copying the code around
  • Dispatch system with safety to disable NLLS with NewtonRaphson (and similarly for others). Currently it will work which is not the worst thing, but disabling this is a safer option IMO
  • Actually functional reinit interface

How close is this to being ready?

Once tests pass and downstream PRs are merged we should be good to go. I will follow this PR up with:

  1. SingleFactorize<....> which can use the iterator interface and recompute_jacobian to Jacobian only when needed.
  2. Multi-Step Methods
  3. Default change to PolyesterForwardDiff similar to SimpleNonlinearSolve

Documentation

  • Dev Docs

Target Issues

Modular Internals

  • Descent
    • Newton -- Automatically handle normal form efficiently
    • Cauchy
    • Dogleg -- Mix Newton and Cauchy
    • Damped Newton -- Yes Levenberg-Marquardt and PseudoTransient can share the same internals
      • Rework a little bit for different schemes
  • Globalization: Only one can be activated at a time
    • Line Search
    • Trust Region

@avik-pal avik-pal force-pushed the ap/rework_internals branch 2 times, most recently from 7bf098a to 3f58736 Compare January 4, 2024 22:12
Copy link

codecov bot commented Jan 4, 2024

Codecov Report

Attention: 310 lines in your changes are missing coverage. Please review.

Comparison is base (b661754) 83.79% compared to head (1d7d202) 86.23%.

Files Patch % Lines
src/core/approximate_jacobian.jl 77.27% 35 Missing ⚠️
src/globalization/trust_region.jl 84.50% 31 Missing ⚠️
src/globalization/line_search.jl 83.94% 22 Missing ⚠️
src/utils.jl 70.83% 21 Missing ⚠️
src/internal/approximate_initialization.jl 85.27% 19 Missing ⚠️
src/descent/damped_newton.jl 88.02% 17 Missing ⚠️
src/algorithms/lbroyden.jl 78.37% 16 Missing ⚠️
src/internal/operators.jl 86.11% 15 Missing ⚠️
src/algorithms/levenberg_marquardt.jl 80.82% 14 Missing ⚠️
src/descent/geodesic_acceleration.jl 71.42% 14 Missing ⚠️
... and 21 more
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #345      +/-   ##
==========================================
+ Coverage   83.79%   86.23%   +2.44%     
==========================================
  Files          28       44      +16     
  Lines        2179     2609     +430     
==========================================
+ Hits         1826     2250     +424     
- Misses        353      359       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@termi-official
Copy link

termi-official commented Jan 5, 2024

I may have a special case here to consider and will write it here instead of as a separate issue due to the title of the PR. If I should transfer this information to an issue instead, I am happy to do it.

When discretizing PDEs we often end up with a nonlinear system of equations + constraints (coming from e.g. Dirichlet boundary conditions). For example in Newton methods we also know how to enforce the constraints efficiently: We can rewrite the linear equation system. In Ferrite.jl a typical Newton-Raphson implementation looks like this:

dh = ... # Holds info about finite element problem
un = zeros(ndofs(dh))
u = zeros(ndofs(dh))
Δu = zeros(ndofs(dh))
... # define remaining variables
dbcs = ... # Holds Dirichlet boundary condition information 
apply!(un, dbcs) # Apply Dirichlet conditions to solution vector
while true; newton_itr += 1
        # Construct the current guess
        u .= un .+ Δu
        # Compute residual and tangent for current guess
        assemble_global!(K, g, u, ...)
        # Apply boundary conditions
        apply_zero!(K, g, dbcs)
        # Compute the residual norm and compare with tolerance
        normg = norm(g)
        if normg < NEWTON_TOL
            break
        elseif newton_itr > NEWTON_MAXITER
            error("Reached maximum Newton iterations, aborting")
        end

        # Compute increment
        ΔΔu =  K \ g

        apply_zero!(ΔΔu, dbcs)
        Δu .-= ΔΔu
    end

(see https://ferrite-fem.github.io/Ferrite.jl/dev/tutorials/hyperelasticity/ for full example).

I am confident that other frameworks come with similar mechanisms. Basically the difference to a "normal" Newton-Raphson is the apply! before entering the iteration to e.g. set the boundary condition in the solution vector and the 2x apply_zero!, first to e.g. eliminate the boundary conditions from the linear system and second to remove numerical inaccurracies in the solver from the increment. If it is not clear: The increment must be zero, because the solution is already enforced to be correct by the first call to apply!. For e.g. gradient based solvers we need similar hooks.

With this in mind I would like to ask for considering this and possibly recommendations about how this integration can be accompished from user side when using NonlinearSolve.jl. I am happy to help here and also to answer any questions which might come up.

My first idea here was to provide a custom linear solver. This custom solver just wraps the actual linear solver and adds the apply_zero calls in the solve function (and when the linear system is regenerated). However, in this case it is not clear how to introduce the first apply! in a clean way before going into the iteration.

@avik-pal
Copy link
Member Author

avik-pal commented Jan 5, 2024

One way might be to provide a preamble / postamble functions with the input being cache. I think that might solve your problem @termi-official? Then you can construct the GeneralizedFirstOr.... method with NewtonDescent to get your NewtonRaphson.

The only concern here is that with custom functions, the inputs to the step! like recompute_jacobian might have to be set as incompatible -- unless we rework how the jacobian or fu is stored / cached.

@ChrisRackauckas do you have particular thoughts on this?

@avik-pal avik-pal mentioned this pull request Jan 6, 2024
Comment on lines +26 to +28
function SciMLBase.reinit!(cache::LeastSquaresOptimJLCache, args...; kwargs...)
error("Reinitialization not supported for LeastSquaresOptimJL.")
end
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why only this one but not the other extensions?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the only one that creates the cache. All others directly use __solve

kwargs...) where {F, IN} --> AbstractNonlinearSolveLineSearchCache
```
"""
abstract type AbstractNonlinearSolveLineSearchAlgorithm end
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wasn't this moved to LineSearch.jl?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not yet, but that is the eventual plan. We still need to figure out how to decouple the line searches like RobustNonMonotoneLineSearch which heavily ties into the cache.

src/default.jl Outdated Show resolved Hide resolved
if is_extension_loaded(Val(:Symbolics))
return SymbolicsSparsityDetection()
else
return ApproximateJacobianSparsity()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a default?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Symbolics is actually pretty good after a size threshold, and even faster than the approximate version.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean defaulting to approximate Jacobian sparsity seems a bit odd. Even if it's not AD?

@ChrisRackauckas
Copy link
Member

A bunch of small comments and issues to open, otherwise good to go!

@avik-pal avik-pal mentioned this pull request Jan 16, 2024
3 tasks
@avik-pal
Copy link
Member Author

the dae failure is real https://github.com/SciML/NonlinearSolve.jl/actions/runs/7543078285/job/20533293590?pr=345#step:6:1759. The cache is initialized with a Vector but later it gets a subarray

Copy link
Member

@ChrisRackauckas ChrisRackauckas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like my comments were addressed. Let's get some issues open on nonlinear preconditioning, Klement in the polyalg, linesearch removal, and non-scalar tolerances.

@ChrisRackauckas
Copy link
Member

the dae failure is real https://github.com/SciML/NonlinearSolve.jl/actions/runs/7543078285/job/20533293590?pr=345#step:6:1759. The cache is initialized with a Vector but later it gets a subarray

We should just make the cache via a view too.

@ChrisRackauckas ChrisRackauckas merged commit 1c427a6 into SciML:master Jan 17, 2024
24 of 26 checks passed
@ChrisRackauckas
Copy link
Member

Make an issue in OrdinaryDiffEq with the DAE one, I can take a look at that later today.

@avik-pal avik-pal deleted the ap/rework_internals branch April 12, 2024 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Normal Form Linear Solve in Trust Region Methods Can not iterate solver before solving
3 participants