Skip to content

Change in behavior for reinterpreted arrays between Julia 1.10 and Julia 1.11 #2382

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jClugstor opened this issue May 5, 2025 · 11 comments

Comments

@jClugstor
Copy link

jClugstor commented May 5, 2025

I'm getting an issue where the behavior around reinterpreted arrays is different between Julia 1.10 and Julia 1.11.

Julia 1.11:

Julia Version 1.11.3
Commit d63adeda50d (2025-01-21 19:42 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 16 × AMD Ryzen 7 6800U with Radeon Graphics
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 16 virtual cores)
Pkg> st
  [7da242da] Enzyme v0.13.41
⌅ [f6369f11] ForwardDiff v0.10.38
  [1dea7af3] OrdinaryDiffEq v6.95.1
using Enzyme
using ForwardDiff
using OrdinaryDiffEq
p = rand(3)

function dudt(u, p, t)
    u .* p
end

f = ODEFunction(dudt)

ytmp = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
tmp1 = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
fx_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
x_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))

pf = let f = f
    function (out, u, _p, t)
        out .= f(u, _p, t)
        nothing
    end
end
_tmp6 = pf

dup = Enzyme.Duplicated(p, [0.0, 0.0, 0.0])

Enzyme.autodiff(Enzyme.Reverse, Enzyme.Duplicated(pf, _tmp6),
    Enzyme.Const, Enzyme.Duplicated(fx_dual, x_dual),
    Enzyme.Duplicated(ytmp, tmp1),
    dup, Enzyme.Const(0.0))

gives

ERROR: LLVM error: function failed verification (4)
Stacktrace:
  [1] handle_error(reason::Cstring)
    @ LLVM ~/.julia/packages/LLVM/xTJfF/src/core/context.jl:194
  [2] EnzymeCreatePrimalAndGradient(logic::Enzyme.Logic, todiff::LLVM.Function, retType::Enzyme.API.CDIFFE_TYPE, constant_args::Vector{…}, TA::Enzyme.TypeAnalysis, returnValue::Bool, dretUsed::Bool, mode::Enzyme.API.CDerivativeMode, runtimeActivity::Bool, width::Int64, additionalArg::Ptr{…}, forceAnonymousTape::Bool, typeInfo::Enzyme.FnTypeInfo, uncacheable_args::Vector{…}, augmented::Ptr{…}, atomicAdd::Bool)
    @ Enzyme.API ~/.julia/packages/Enzyme/3VNOP/src/api.jl:269
  [3] enzyme!(job::GPUCompiler.CompilerJob{…}, mod::LLVM.Module, primalf::LLVM.Function, TT::Type, mode::Enzyme.API.CDerivativeMode, width::Int64, parallel::Bool, actualRetType::Type, wrap::Bool, modifiedBetween::NTuple{…} where N, returnPrimal::Bool, expectedTapeType::Type, loweredArgs::Set{…}, boxedArgs::Set{…})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:1754
  [4] codegen(output::Symbol, job::GPUCompiler.CompilerJob{…}; libraries::Bool, deferred_codegen::Bool, optimize::Bool, toplevel::Bool, strip::Bool, validate::Bool, only_entry::Bool, parent_job::Nothing)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:4664
  [5] codegen
    @ ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:3450 [inlined]
  [6] _thunk(job::GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget, Enzyme.Compiler.EnzymeCompilerParams}, postopt::Bool)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:5528
  [7] _thunk
    @ ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:5528 [inlined]
  [8] cached_compilation
    @ ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:5580 [inlined]
  [9] thunkbase(mi::Core.MethodInstance, World::UInt64, FA::Type{…}, A::Type{…}, TT::Type, Mode::Enzyme.API.CDerivativeMode, width::Int64, ModifiedBetween::NTuple{…} where N, ReturnPrimal::Bool, ShadowInit::Bool, ABI::Type, ErrIfFuncWritten::Bool, RuntimeActivity::Bool, edges::Vector{…})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:5691
 [10] thunk_generator(world::UInt64, source::LineNumberNode, FA::Type, A::Type, TT::Type, Mode::Enzyme.API.CDerivativeMode, Width::Int64, ModifiedBetween::NTuple{…} where N, ReturnPrimal::Bool, ShadowInit::Bool, ABI::Type, ErrIfFuncWritten::Bool, RuntimeActivity::Bool, self::Any, fakeworld::Any, fa::Type, a::Type, tt::Type, mode::Type, width::Type, modifiedbetween::Type, returnprimal::Type, shadowinit::Type, abi::Type, erriffuncwritten::Type, runtimeactivity::Type)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:5876
 [11] autodiff(::ReverseMode{…}, ::Duplicated{…}, ::Type{…}, ::Duplicated{…}, ::Duplicated{…}, ::Duplicated{…}, ::Const{…})
    @ Enzyme ~/.julia/packages/Enzyme/3VNOP/src/Enzyme.jl:485
 [12] top-level scope
    @ REPL[72]:1
Some type information was truncated. Use `show(err)` to see complete types.

If you do this (note the upper case Nothings):

using Enzyme
using ForwardDiff
using OrdinaryDiffEq
p = rand(3)

function dudt(u, p, t)
    u .* p
end

f = ODEFunction(dudt)

ytmp = reinterpret(ForwardDiff.Dual{Nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
tmp1 = reinterpret(ForwardDiff.Dual{Nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
fx_dual = reinterpret(ForwardDiff.Dual{Nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
x_dual = reinterpret(ForwardDiff.Dual{Nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))

pf = let f = f
    function (out, u, _p, t)
        out .= f(u, _p, t)
        nothing
    end
end
_tmp6 = pf

dup = Enzyme.Duplicated(p, [0.0, 0.0, 0.0])

Enzyme.autodiff(Enzyme.Reverse, Enzyme.Duplicated(pf, _tmp6),
    Enzyme.Const, Enzyme.Duplicated(fx_dual, x_dual),
    Enzyme.Duplicated(ytmp, tmp1),
    dup, Enzyme.Const(0.0))

it works fine. The only difference is that ytmp etc. are no longer ReinterpretedArrays. The thing is, the first one works on Julia 1.10:

Julia Version 1.10.9
Commit 5595d20a287 (2025-03-10 12:51 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 16 × AMD Ryzen 7 6800U with Radeon Graphics
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 16 virtual cores)
(OnePointTen) pkg> st
Status `~/Documents/Work/dev/ODETesting/SciMLSensitivity/OnePointTen/Project.toml`
  [7da242da] Enzyme v0.13.41
⌅ [f6369f11] ForwardDiff v0.10.38
  [1dea7af3] OrdinaryDiffEq v6.95.1

You can run this without problems,

using Enzyme
using ForwardDiff
using OrdinaryDiffEq
p = rand(3)

function dudt(u, p, t)
    u .* p
end

f = ODEFunction(dudt)

ytmp = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
tmp1 = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
fx_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
x_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))

pf = let f = f
    function (out, u, _p, t)
        out .= f(u, _p, t)
        nothing
    end
end
_tmp6 = pf

dup = Enzyme.Duplicated(p, [0.0, 0.0, 0.0])

Enzyme.autodiff(Enzyme.Reverse, Enzyme.Duplicated(pf, _tmp6),
    Enzyme.Const, Enzyme.Duplicated(fx_dual, x_dual),
    Enzyme.Duplicated(ytmp, tmp1),
    dup, Enzyme.Const(0.0))

One thing is that this:

using Enzyme
using ForwardDiff

struct FunctionHolder
    f
end

(f::FunctionHolder)(args...) = f.f(args...)

f = FunctionHolder(dudt)

tmp = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
tmp1 = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
fx_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
x_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))

pf = let f = f
    function (out, u, _p, t)
        out .= f(u, _p, t)
        nothing
    end
end
_tmp6 = pf

dup = Enzyme.Duplicated(p, [0.0, 0.0, 0.0])

Enzyme.autodiff(Enzyme.Reverse, Enzyme.Duplicated(pf, _tmp6),
    Enzyme.Const, Enzyme.Duplicated(fx_dual, x_dual),
    Enzyme.Duplicated(ytmp, tmp1),
    dup, Enzyme.Const(0.0))

works perfectly fine on Julia 1.11, even though they should essentially be doing the exact same thing I think.

Another thing is that if you specify SciMLBase.NoSpecialize that gets rid of the error as well:

using Enzyme
using ForwardDiff
using OrdinaryDiffEq
using SciMLBase
p = rand(3)

function dudt(u, p, t)
    u .* p
end

f = ODEFunction{false, SciMLBase.NoSpecialize}(dudt)

ytmp = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
tmp1 = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
fx_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
x_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))

pf = let f = f
    function (out, u, _p, t)
        out .= f.f(u, _p, t)
        nothing
    end
end
_tmp6 = pf

dup = Enzyme.Duplicated(p, [0.0, 0.0, 0.0])

Enzyme.autodiff(Enzyme.Reverse, Enzyme.Duplicated(pf, _tmp6),
    Enzyme.Const, Enzyme.Duplicated(fx_dual, x_dual),
    Enzyme.Duplicated(ytmp, tmp1),
    dup, Enzyme.Const(0.0))
@jClugstor

This comment has been minimized.

@jClugstor
Copy link
Author

This one is related to the Core 2 SciMLSensitivity tests, https://github.com/SciML/SciMLSensitivity.jl/actions/runs/14969197208/job/42046022185

@ChrisRackauckas
Copy link
Contributor

We one is deprioritized a bit from the others since we found a viable workaround SciML/SciMLSensitivity.jl#1191

@wsmoses
Copy link
Member

wsmoses commented May 12, 2025

 Type of ghost or constant type Duplicated{var"#37#38"{FunctionHolder{typeof(dudt)}}} is marked as differentiable.

this one is a user-side error, you're marking a non-differentiable type as differentiable

@jClugstor
Copy link
Author

Sorry, probably shouldn't have conflated those, I just wasn't sure how it was actually different from the other example and I was trying to find a simpler working example.

The actual error is the ERROR: LLVM error: function failed verification (4) from the example at the top.

@wsmoses
Copy link
Member

wsmoses commented May 12, 2025

ok cool, well in any case if you can make a more minimal example that has the same error (equally applies for all the other issues), it'll allow us to get these fixed faster

@jClugstor
Copy link
Author

Yeah for sure, the example at the very top is the result of many hours of figuring out exactly how SciMLSensitivity calls Enzyme with what types and trying to recreate it. For some reason I can't get the exact same error message without using ODEFunction, so that's about as simple as I can seem to make it unfortunately.

@wsmoses
Copy link
Member

wsmoses commented May 12, 2025

perhaps cc @oscardssmith for support

@oscardssmith
Copy link
Contributor

this is probably Enzyme strugling with FunctionWrappersWrappers

@ChrisRackauckas
Copy link
Contributor

No SciMLSensitivity unwraps.

@jClugstor
Copy link
Author

Yeah they get unwrapped, none of the functions in the MWE are FunctionWrappers or FunctionWrapperWrappers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants