Change in behavior for reinterpreted arrays between Julia 1.10 and Julia 1.11 #2382

jClugstor · 2025-05-05T17:55:57Z

I'm getting an issue where the behavior around reinterpreted arrays is different between Julia 1.10 and Julia 1.11.

Julia 1.11:

Julia Version 1.11.3
Commit d63adeda50d (2025-01-21 19:42 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 16 × AMD Ryzen 7 6800U with Radeon Graphics
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 16 virtual cores)

Pkg> st
  [7da242da] Enzyme v0.13.41
⌅ [f6369f11] ForwardDiff v0.10.38
  [1dea7af3] OrdinaryDiffEq v6.95.1

using Enzyme
using ForwardDiff
using OrdinaryDiffEq
p = rand(3)

function dudt(u, p, t)
    u .* p
end

f = ODEFunction(dudt)

ytmp = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
tmp1 = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
fx_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
x_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))

pf = let f = f
    function (out, u, _p, t)
        out .= f(u, _p, t)
        nothing
    end
end
_tmp6 = pf

dup = Enzyme.Duplicated(p, [0.0, 0.0, 0.0])

Enzyme.autodiff(Enzyme.Reverse, Enzyme.Duplicated(pf, _tmp6),
    Enzyme.Const, Enzyme.Duplicated(fx_dual, x_dual),
    Enzyme.Duplicated(ytmp, tmp1),
    dup, Enzyme.Const(0.0))

gives

ERROR: LLVM error: function failed verification (4)
Stacktrace:
  [1] handle_error(reason::Cstring)
    @ LLVM ~/.julia/packages/LLVM/xTJfF/src/core/context.jl:194
  [2] EnzymeCreatePrimalAndGradient(logic::Enzyme.Logic, todiff::LLVM.Function, retType::Enzyme.API.CDIFFE_TYPE, constant_args::Vector{…}, TA::Enzyme.TypeAnalysis, returnValue::Bool, dretUsed::Bool, mode::Enzyme.API.CDerivativeMode, runtimeActivity::Bool, width::Int64, additionalArg::Ptr{…}, forceAnonymousTape::Bool, typeInfo::Enzyme.FnTypeInfo, uncacheable_args::Vector{…}, augmented::Ptr{…}, atomicAdd::Bool)
    @ Enzyme.API ~/.julia/packages/Enzyme/3VNOP/src/api.jl:269
  [3] enzyme!(job::GPUCompiler.CompilerJob{…}, mod::LLVM.Module, primalf::LLVM.Function, TT::Type, mode::Enzyme.API.CDerivativeMode, width::Int64, parallel::Bool, actualRetType::Type, wrap::Bool, modifiedBetween::NTuple{…} where N, returnPrimal::Bool, expectedTapeType::Type, loweredArgs::Set{…}, boxedArgs::Set{…})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:1754
  [4] codegen(output::Symbol, job::GPUCompiler.CompilerJob{…}; libraries::Bool, deferred_codegen::Bool, optimize::Bool, toplevel::Bool, strip::Bool, validate::Bool, only_entry::Bool, parent_job::Nothing)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:4664
  [5] codegen
    @ ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:3450 [inlined]
  [6] _thunk(job::GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget, Enzyme.Compiler.EnzymeCompilerParams}, postopt::Bool)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:5528
  [7] _thunk
    @ ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:5528 [inlined]
  [8] cached_compilation
    @ ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:5580 [inlined]
  [9] thunkbase(mi::Core.MethodInstance, World::UInt64, FA::Type{…}, A::Type{…}, TT::Type, Mode::Enzyme.API.CDerivativeMode, width::Int64, ModifiedBetween::NTuple{…} where N, ReturnPrimal::Bool, ShadowInit::Bool, ABI::Type, ErrIfFuncWritten::Bool, RuntimeActivity::Bool, edges::Vector{…})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:5691
 [10] thunk_generator(world::UInt64, source::LineNumberNode, FA::Type, A::Type, TT::Type, Mode::Enzyme.API.CDerivativeMode, Width::Int64, ModifiedBetween::NTuple{…} where N, ReturnPrimal::Bool, ShadowInit::Bool, ABI::Type, ErrIfFuncWritten::Bool, RuntimeActivity::Bool, self::Any, fakeworld::Any, fa::Type, a::Type, tt::Type, mode::Type, width::Type, modifiedbetween::Type, returnprimal::Type, shadowinit::Type, abi::Type, erriffuncwritten::Type, runtimeactivity::Type)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:5876
 [11] autodiff(::ReverseMode{…}, ::Duplicated{…}, ::Type{…}, ::Duplicated{…}, ::Duplicated{…}, ::Duplicated{…}, ::Const{…})
    @ Enzyme ~/.julia/packages/Enzyme/3VNOP/src/Enzyme.jl:485
 [12] top-level scope
    @ REPL[72]:1
Some type information was truncated. Use `show(err)` to see complete types.

If you do this (note the upper case Nothings):

using Enzyme
using ForwardDiff
using OrdinaryDiffEq
p = rand(3)

function dudt(u, p, t)
    u .* p
end

f = ODEFunction(dudt)

ytmp = reinterpret(ForwardDiff.Dual{Nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
tmp1 = reinterpret(ForwardDiff.Dual{Nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
fx_dual = reinterpret(ForwardDiff.Dual{Nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
x_dual = reinterpret(ForwardDiff.Dual{Nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))

pf = let f = f
    function (out, u, _p, t)
        out .= f(u, _p, t)
        nothing
    end
end
_tmp6 = pf

dup = Enzyme.Duplicated(p, [0.0, 0.0, 0.0])

Enzyme.autodiff(Enzyme.Reverse, Enzyme.Duplicated(pf, _tmp6),
    Enzyme.Const, Enzyme.Duplicated(fx_dual, x_dual),
    Enzyme.Duplicated(ytmp, tmp1),
    dup, Enzyme.Const(0.0))

it works fine. The only difference is that ytmp etc. are no longer ReinterpretedArrays. The thing is, the first one works on Julia 1.10:

Julia Version 1.10.9
Commit 5595d20a287 (2025-03-10 12:51 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 16 × AMD Ryzen 7 6800U with Radeon Graphics
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 16 virtual cores)

(OnePointTen) pkg> st
Status `~/Documents/Work/dev/ODETesting/SciMLSensitivity/OnePointTen/Project.toml`
  [7da242da] Enzyme v0.13.41
⌅ [f6369f11] ForwardDiff v0.10.38
  [1dea7af3] OrdinaryDiffEq v6.95.1

You can run this without problems,

using Enzyme
using ForwardDiff
using OrdinaryDiffEq
p = rand(3)

function dudt(u, p, t)
    u .* p
end

f = ODEFunction(dudt)

ytmp = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
tmp1 = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
fx_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
x_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))

pf = let f = f
    function (out, u, _p, t)
        out .= f(u, _p, t)
        nothing
    end
end
_tmp6 = pf

dup = Enzyme.Duplicated(p, [0.0, 0.0, 0.0])

Enzyme.autodiff(Enzyme.Reverse, Enzyme.Duplicated(pf, _tmp6),
    Enzyme.Const, Enzyme.Duplicated(fx_dual, x_dual),
    Enzyme.Duplicated(ytmp, tmp1),
    dup, Enzyme.Const(0.0))

One thing is that this:

using Enzyme
using ForwardDiff

struct FunctionHolder
    f
end

(f::FunctionHolder)(args...) = f.f(args...)

f = FunctionHolder(dudt)

tmp = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
tmp1 = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
fx_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
x_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))

pf = let f = f
    function (out, u, _p, t)
        out .= f(u, _p, t)
        nothing
    end
end
_tmp6 = pf

dup = Enzyme.Duplicated(p, [0.0, 0.0, 0.0])

Enzyme.autodiff(Enzyme.Reverse, Enzyme.Duplicated(pf, _tmp6),
    Enzyme.Const, Enzyme.Duplicated(fx_dual, x_dual),
    Enzyme.Duplicated(ytmp, tmp1),
    dup, Enzyme.Const(0.0))

works perfectly fine on Julia 1.11, even though they should essentially be doing the exact same thing I think.

Another thing is that if you specify SciMLBase.NoSpecialize that gets rid of the error as well:

using Enzyme
using ForwardDiff
using OrdinaryDiffEq
using SciMLBase
p = rand(3)

function dudt(u, p, t)
    u .* p
end

f = ODEFunction{false, SciMLBase.NoSpecialize}(dudt)

ytmp = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
tmp1 = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
fx_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
x_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))

pf = let f = f
    function (out, u, _p, t)
        out .= f.f(u, _p, t)
        nothing
    end
end
_tmp6 = pf

dup = Enzyme.Duplicated(p, [0.0, 0.0, 0.0])

Enzyme.autodiff(Enzyme.Reverse, Enzyme.Duplicated(pf, _tmp6),
    Enzyme.Const, Enzyme.Duplicated(fx_dual, x_dual),
    Enzyme.Duplicated(ytmp, tmp1),
    dup, Enzyme.Const(0.0))

The text was updated successfully, but these errors were encountered:

jClugstor · 2025-05-12T14:12:38Z

This one is related to the Core 2 SciMLSensitivity tests, https://github.com/SciML/SciMLSensitivity.jl/actions/runs/14969197208/job/42046022185

ChrisRackauckas · 2025-05-12T14:15:39Z

We one is deprioritized a bit from the others since we found a viable workaround SciML/SciMLSensitivity.jl#1191

wsmoses · 2025-05-12T18:35:08Z

 Type of ghost or constant type Duplicated{var"#37#38"{FunctionHolder{typeof(dudt)}}} is marked as differentiable.

this one is a user-side error, you're marking a non-differentiable type as differentiable

jClugstor · 2025-05-12T18:57:08Z

Sorry, probably shouldn't have conflated those, I just wasn't sure how it was actually different from the other example and I was trying to find a simpler working example.

The actual error is the ERROR: LLVM error: function failed verification (4) from the example at the top.

wsmoses · 2025-05-12T19:21:41Z

ok cool, well in any case if you can make a more minimal example that has the same error (equally applies for all the other issues), it'll allow us to get these fixed faster

jClugstor · 2025-05-12T20:08:07Z

Yeah for sure, the example at the very top is the result of many hours of figuring out exactly how SciMLSensitivity calls Enzyme with what types and trying to recreate it. For some reason I can't get the exact same error message without using ODEFunction, so that's about as simple as I can seem to make it unfortunately.

wsmoses · 2025-05-12T20:10:37Z

perhaps cc @oscardssmith for support

oscardssmith · 2025-05-12T21:20:28Z

this is probably Enzyme strugling with FunctionWrappersWrappers

ChrisRackauckas · 2025-05-12T21:45:52Z

No SciMLSensitivity unwraps.

jClugstor · 2025-05-12T21:54:58Z

Yeah they get unwrapped, none of the functions in the MWE are FunctionWrappers or FunctionWrapperWrappers.

This comment has been minimized.

Sign in to view

ChrisRackauckas mentioned this issue May 5, 2025

Enzyme internal error while running neural ODE with Lux + Enzyme #2110

Open

jClugstor mentioned this issue May 12, 2025

Slight regression caused by using LazyBufferCache in EnzymeVJP SciML/SciMLSensitivity.jl#1197

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Change in behavior for reinterpreted arrays between Julia 1.10 and Julia 1.11 #2382

Change in behavior for reinterpreted arrays between Julia 1.10 and Julia 1.11 #2382

jClugstor commented May 5, 2025 •

edited

Loading

This comment has been minimized.

jClugstor commented May 12, 2025

Uh oh!

ChrisRackauckas commented May 12, 2025

Uh oh!

wsmoses commented May 12, 2025

Uh oh!

jClugstor commented May 12, 2025

Uh oh!

wsmoses commented May 12, 2025

Uh oh!

jClugstor commented May 12, 2025

Uh oh!

wsmoses commented May 12, 2025

Uh oh!

oscardssmith commented May 12, 2025

Uh oh!

ChrisRackauckas commented May 12, 2025

Uh oh!

jClugstor commented May 12, 2025

Uh oh!

Change in behavior for reinterpreted arrays between Julia 1.10 and Julia 1.11 #2382

Change in behavior for reinterpreted arrays between Julia 1.10 and Julia 1.11 #2382

Comments

jClugstor commented May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This comment has been minimized.

jClugstor commented May 12, 2025

Uh oh!

ChrisRackauckas commented May 12, 2025

Uh oh!

wsmoses commented May 12, 2025

Uh oh!

jClugstor commented May 12, 2025

Uh oh!

wsmoses commented May 12, 2025

Uh oh!

jClugstor commented May 12, 2025

Uh oh!

wsmoses commented May 12, 2025

Uh oh!

oscardssmith commented May 12, 2025

Uh oh!

ChrisRackauckas commented May 12, 2025

Uh oh!

jClugstor commented May 12, 2025

Uh oh!

jClugstor commented May 5, 2025 •

edited

Loading