ForwardDiff.jl Usage

[ChrisRackauckas]

I think one thing that needs to be discussed on the roadmap is how to use ForwardDiff.jl. DiffEq has a long not so happy relationship with ForwardDiff.jl. It's not the developers' fault, it's moreso that our usage doesn't map to their API well.

Two examples. First, directly calculating derivatives and gradients. Our standard functional form is f(t,u,du), which is inplace u' = f(t,u). There are other forms, but this one matters most and if we get this, we get the rest. Here, t is a scalar (time) and u is a vector. du is a vector for the output. This pre-allocated form works well for DiffEqs because it's standard (all of the C/Fortran solvers use this form as well). However, ForwardDiff, nor Calculus/FiniteDiff accept this form.

Thus there is a lot of code to force it to work. For example, as explained a bit in #1, every function gets wrapped in a callable type and only exposes the function arguments ForwardDiff wishes to see:

https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/master/src/derivative_wrappers.jl

It also turns all arrays into vectors, since there was a problem (at least there was before). But this causes a problem where we have to make sure that these internal caches are defined appropriately, both duals or not and matching chunk sizes. So there's a whole other file for handling that:

https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/master/src/misc_utils.jl

Then the internal caches for the diffeq need to fuss with this issue:

https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/master/src/caches.jl#L995

and finally, the internal caches in the functions need to all be updated before each derivative/Jacobian call

https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/master/src/integrators/rosenbrock_integrators.jl#L17

That's the easy case. The harder case is using NLsolve. It was brought up here:

JuliaDiff/ForwardDiff.jl#136

The solution uses the ducache

https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/master/src/misc_utils.jl#L1

which for example turns implicit Euler into

https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/master/src/integrators/implicit_integrators.jl#L47

Needless to say, this is definitely the most convoluted part of DiffEq, which is why the derivative tooling needs its own package.

But is there a better solution? Should we be using Dual numbers more directly? Or does the recent change to ForwardDiff

JuliaDiff/ForwardDiff.jl#213

make it possible that supporting derivative! in time on f(t,u,du) can be more direct? I am interested in hearing your thoughts @jrevels.

[ChrisRackauckas]

Related ForwardDiff problems is lack of complex support:

SciML/DifferentialEquations.jl#110

I'd like to see if that could be tackled simultaneously.