Derivative of inplace functions

[ChrisRackauckas]

There is a jacobian! for f!(y,x), but is there no derivative! for f(y,t) where t is a scalar?

[KristofferC]

How can you make an in place version of something that returns a scalar?

[ChrisRackauckas]

No it takes in a scalar and returns a vector...

…

On Apr 20, 2017 10:43 PM, "Kristoffer Carlsson" ***@***.***> wrote: How can you make an in place version of something that returns a scalar? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#218 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABuunrx_Na6D1NbnyFJ2fY8FiHrLQXv4ks5ryEH1gaJpZM4NDxwO> .

[KristofferC]

Hm, I thought derivative was only for scalar -> scalar but apparently not. You could solve this with a f(x::Vector) = f(x[1]) and use the jacobian API. But yeah, it seems that this functionality is missing.

[jrevels]

I agree, this would be an appropriate addition to the API.

@ChrisRackauckas Would this fix JuliaDiff/FiniteDiff.jl#3, since you could then do derivative!(dt, (du, t) -> f(t, u, du), du, t)?

[ChrisRackauckas]

It doesn't fix all of JuliaDiff/FiniteDiff.jl#3, but it would quite a bit.

derivative!(dt, (du, t) -> f(t, u, du), du, t)

If this is in a loop, is it generating the closure every time? That's why I am using call-overloaded types and then just making the u a variable in the type. Small example of how this is done:

# Setup
type VectorF{F,SizeType,uType,rateType} <: Function
  f::F
  u::uType
  du::rateType
end
function (p::VectorF)(t)
  p.f(t,reshape(uprev,size(p.u)),reshape(du,size(p.u)))
  du = vec(du)
end
# Now build the function that can be differentiated
# Note that `du` is a cache that needs to be typed for duals
f_diff = VectorF(f,some_u_cache,du)

# Usage
for i in ...
  f_diff.u .= u # update internal cache
  derivative!(dt,f_diff,t)
end

I think that since the time I wrote this, the reshaping is now done internally by ForwardDiff? (I found a commit that added a bunch of reshapes, but just making sure). Is that true for derivatives as well? So if Julia automatically sets it up to only compile (du, t) -> f(t, u, du) once instead of in a loop, plus this in place form is added, that would indeed get rid of all of this extra code.

I need to test this closure usage more, and also see that if the closure is behind a function yet in the loop if it also would get replaced.

[KristofferC]

Anonymous functions are only "generated" once and are given a unique type at creation. They should lower to something similar that you have done manually here. Always gotta worry about the Boxes though..

[ChrisRackauckas]

Anonymous functions are only "generated" once and are given a unique type at creation.

That's good to know. So then if ForwardDiff does the vec and reshape internally for derivative! and jacobian!, and derivative! gets this in-place format, I think all of the code from

https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/master/src/derivative_wrappers.jl

Can be deleted and we can do straight calls to ForwadDiff.jl, which would make me more than happy.

This doesn't fix the usage of NLsolve + autodifferentiation + inplace functions (#136), but I am not sure that has an elegant fix and will require specialty code.

Always gotta worry about the Boxes though..

Yes, I think before I ran into JuliaLang/julia#15276.

[jrevels]

I think that since the time I wrote this, the reshaping is now done internally by ForwardDiff? (I found a commit that added a bunch of reshapes, but just making sure). Is that true for derivatives as well?

Right, arrays of any shape should be usable now. IIRC, this was always the case for derivative, which shouldn't require any reshaping (unless I'm remembering wrong, the main issue was generalizing jacobian).

If this is in a loop, is it generating the closure every time? That's why I am using call-overloaded types and then just making the u a variable in the type.

Like Kristoffer said, the anonymous function is lowered into a callable type where the fields correspond to closed-over variables. In practice, Julia will sometimes box these fields unnecessarily because it can't determine that they're type-constant, so I've taken to defining a callable struct manually just for piece of mind. Also keep in mind that instantiating a callable type which contains pointers to heap-allocated objects may cause the callable type instance to itself be heap-allocated, since we don't have a stack-aware GC (probably my biggest pain-point with Julia atm).