New function Base.unsafe_wrap! to avoid unnecessary heap allocations

[MartinOtter]

If a C-function calls many times a Julia function and passes C-arrays, for example, when performing simulation, optimization, model-based controllers etc. (e.g. this case happens with the Sundials integrators, see e.g. issue https://github.com/JuliaDiffEq/Sundials.jl/pull/179), then C-arrays must be wrapped to Julia arrays. This is performed with the existing Base.unsafe_wrap(..) function. However, this function returns a Julia array and therefore allocates heap memory. In a typical simulation with Sundials IDA or CVODE this happens, say, 1000 - 10000 times for 3 arrays leading to many small heap allocations that dominate the function evaluation cost for smaller models. This performance issue could be fixed, if the C-array pointer could be replaced in an unsafe_wrap(..)ed array:

unsafe_wrap!(array, pointer::Ptr{T})

It is assumed that array is an existing Julia array generated with the unsafe_wrap(..) function with keyword argument own=false. Function unsafe_wrap!(..) changes the existing pointer to the array data to argument pointer. Hereby, it is assumed that the length of the data remains the same.

[yuyichao]

This will break many assumptions.

then C-arrays must be wrapped to Julia arrays.

No, you can use a custom array type.

[MartinOtter]

Sorry, if I overlooked this. I searched, but did not yet find a way, how a C-pointer (that is changed for every call of the Julia function) could be used as pointer in a Julia array without function unsafe_wrap!. Can you give some additional hints how a "custom array type" could be defined in this way (note, on the C-side an existing DLL is used and it is not possible to (practically) change the C-code).

[yuyichao]

used as pointer in a Julia array

You don't. At least not as an Array.

how a "custom array type" could be defined

There are countless number of examples here. https://github.com/JuliaArrays. You should even be able to tweak https://github.com/JuliaArrays/UnalignedVectors.jl to fit your need easily.

[MartinOtter]

Thanks, I understand now: A new custom array type has to be constructed using functions Base.unsafe_load and Base.unsafe_store! to access/store elements of the array given a C-pointer.
Closing this issue.

[ChrisRackauckas]

That doesn't entirely fix the issue since then the array is missing all of the standard overloads like matrix multiplication. Specifically for Sundials, the pointers already have a compatible array via the NVector type which matches the Sundials NVector https://github.com/JuliaDiffEq/Sundials.jl/blob/master/src/nvector_wrapper.jl , but relying on only getindex/setindex! does reduce the available functionality.

[MartinOtter]

Reopened, due to the comment above

[Keno]

Have you considered allocating the memory in julia and using that memory as the backing store on the c side instead, rather than trying to shove a C-managed pointer into a julia array? I agree with @yuyichao that for the latter case, a custom array type would work better. Even if you could write an unsafe_wrap! function now, it's unclear that you could continue doing so as we add optimizations to make julia arrays faster.

[ChrisRackauckas]

Have you considered allocating the memory in julia and using that memory as the backing store on the c side instead, rather than trying to shove a C-managed pointer into a julia array?

Yes, but that's exactly what cannot be done in the linked example SciML/Sundials.jl#179 . Sundials has a buffer of arrays that it can give back. This function is given to and called by Sundials:

function cvodefunjac(t::Float64,
                     u::N_Vector,
                     du::N_Vector,
                     funjac::FunJac)
    if !(pointer(funjac.u) === __N_VGetArrayPointer_Serial(u))
      #@warn "Pointer is broken to FunJac.u"
      funjac.u = convert(Vector, u)
    end
    _u = funjac.u

    if !(pointer(funjac.du) === __N_VGetArrayPointer_Serial(du))
      #@warn "Pointer is broken to FunJac.du"
      funjac.du = convert(Vector, du)
    end
    _du = funjac.du

    funjac.fun(_du, _u, funjac.p, t)
    return CV_SUCCESS
end

The arrays Sundials has are allocated in Julia and then given to Sundials. But then CVODE can give back any of its internal arrays and it seems that these change whenever it changes the order of the method (adaptive order), so funjac.u keeps changing because it can keep changing the arrays. We can do something where we store all of Sundials' possible arrays and then check pointers, or do an unsafe_copyto! to a Julia array (adding more computations but reduces allocations in an inner loop), but it seemed like the real answer would be to directly fix this.

The performance issue was pretty drastic in some tested cases. For a different Sundials algorithm IDA, it only gives the users back the pointers the user used to init, so this same trick in this case made there be zero allocations and resulting in a quite large speedup.

[yuyichao]

the array is missing all of the standard overloads like matrix multiplication

That just says that the array implementation isn't complete and isn't specific to the array implementation you need. With this argument you can say that any array type other than Array are useless, which is clearly not the case.

It's much more helpful to just see which operations are harder to implement for custom array type than they needed to be and improve on that rather than hacking into the base Array type in a way that breaks it's invariance for a uncommon usecase.

[ChrisRackauckas]

That just says that the array implementation isn't complete and isn't specific to the array implementation you need. With this argument you can say that any array type other than Array are useless, which is clearly not the case.

It's much more helpful to just see which operations are harder to implement for custom array type than they needed to be and improve on that rather than hacking into the base Array type in a way that breaks it's invariance for a uncommon usecase.

Okay fine, in those terms then we want a custom array type AArray which has exactly the same overloads as Array for everything except it has a pointer field that we can change. Is there a nice metaprogramming way to forward every possible array dispatch? This seems like a lot of extra work.

[yuyichao]

in those terms then we want a custom array type AArray which has exactly the same overloads as Array for everything except it has a pointer field

No that's not what you want. You want to make sure that all the functions that makes sense for both Array and your AArray is defined for an abstract type rather than Array, e.g. DenseArray. If it's not doable for a method, it means that the method is using implementation detail of Array that won't be applicable to AArray anyway.

[yuyichao]

If it's not doable for a method

e.g. push!

This seems like a lot of extra work.

No it's not. That's the whole point of abstract type. They are not there just to create a good looking tree for presentation, they are actually useful for writing generic functions that only rely on certain shared properties between types and they should certainly be used that way. There are certainly still cases where it's not expressive enough (yet) which is why more expressive variants like traits could be useful, however, in this case, DenseArray seems to cover most if not all the properties you need.

[ChrisRackauckas]

Looks like we can make a struct which is <:DenseVector that just carries a Ref{N_Vector} with a few overloads and it'll work. The standard AbstractArray interface does a lot, and pointer overloads look like they take care of things like BLAS/LAPACK compatibility. That works well enough. This can be closed.

[MartinOtter]

A final question: In NVector of Sundials "Ref{..}" is used on the C-array pointer. If the C-array pointer is changing, I guess that Ref{..} has to be applied again on the new pointer. However, Ref{..} allocates heap memory (for a vector 32 byte, for a matrix 80 bytes on my machine). So, is there a way to avoid appyling Ref{..} on the changing C-array pointer?

[ChrisRackauckas]

NVector is a Julia abstraction which wraps an N_Vector and an Array to make it easy to allocate N_Vectors and garbage control it. It's not needed here. What comes from Sundials is an N_Vector.

https://github.com/JuliaDiffEq/Sundials.jl/blob/master/src/wrapped_api/types_and_consts.jl#L240
https://github.com/JuliaDiffEq/Sundials.jl/blob/master/src/common_interface/function_types.jl#L15-L18

This is just the C-array pointer. We can just put that in a stack-allocated struct to perform dispatches. SciML/Sundials.jl#180

[yuyichao]

carries a Ref{N_Vector}

Why do you need Ref instead of just having it as a field?

In NVector of Sundials "Ref{..}" is used on the C-array pointer

And the only operation done on that Ref is getindex except the finalizer which could/should be on the object itself instead.

[ChrisRackauckas]

Yes, the old NVector is old "good enough" inherited code which should be rewritten. SciML/Sundials.jl#67 . It's a red herring. Hence SciML/Sundials.jl#180 should be good enough to just drop this.

[MartinOtter]

Closing due to the comments above