Split SciMLBase from DiffEqBase #1

ChrisRackauckas · 2021-01-26T03:46:43Z

There are many reasons for this.

DiffEqBase.jl has grown to be a base between the differential equation solvers, with common tools for nonlinear solvers and callbacks adding a ton of dependencies. Users like @mauro3 noticed this in DiffEqBase is not "lightweight" DiffEqBase.jl#618
A lot of DiffEqBase was becoming unrelated to differential equations. OptimizationProblem, NonlinearProblem, all of those functions, etc. It's a full interface for SciML, not DiffEq which is by now a subset of the full ecosystem. But it needed to play the Base role, which meant that nonlinear problems were defined in the repo carrying all of the dependencies for the ODE solver's callback system.
It started inviting circular dependencies. Some of these problems are pervasive. We need to get the simplest form to GalacticOptim.jl, Quadrature.jl, NonlinearSolve.jl, etc. which are becoming the DiffEq's of their respective regimes (though there's not as much to do in those areas). But this meant NonlinearSolve.jl needed NonlinearProblem, but NonlinearSolve.jl was used in the callbacks, so... it had its own problem type. This gives an issue to a system like MTK which then has no unique type to target.
There's a lot of churn on the DiffEq solver portions of DiffEqBase, but relatively little on the actual interface portions. So it makes sense to let them be more stable.
There's a ton of packages which are @requires in DiffEqBase, which adds to the compile time. This was done because the library was supposed to avoid dependencies, but as the central piece of the differential equation solvers it really couldn't, and so it used required to fake lower dependencies but some of those like ForwardDiff.jl should really be promoted to being a real dependency now that it doesn't have the "idea" of being no-dependency.

For backwards compatibility, DiffEqBase reexports SciMLBase and imports a ton of the abstract types it used to have. This should allow for the migration to take place over time, while still fixing some of the MTK and NonlinearSolve issues that need a fix ASAP.

DiffEqBase will have a companion PR to match it.

There are some remaining issues here though:

I would like to get rid of the LabelledArrays dependency. Is there a good generic to use for the names? nameof?
Is the split at the right level?
I want to drop the Queryverse stuff like TableTraits and IteratorInterfaceExtensions since we have Tables.jl
What should we do about our linear solver interface? Is there a way to fold it into these other problems? I think not, but there's a few things to think about.

Downstream changes to setup:

GalacticOptim should use SciMLBase
NonlinearSolve should use SciMLBase
ModelingToolkit should use SciMLBase
Quadrature should use SciMLBase

@mauro3

There are many reasons for this. 1. DiffEqBase.jl has grown to be a base between the differential equation solvers, with common tools for nonlinear solvers and callbacks adding a ton of dependencies. Users like @mauro3 noticed this in SciML/DiffEqBase.jl#618 2. A lot of DiffEqBase was becoming unrelated to differential equations. OptimizationProblem, NonlinearProblem, all of those functions, etc. It's a full interface for SciML, not DiffEq which is by now a subset of the full ecosystem. But it needed to play the Base role, which meant that nonlinear problems were defined in the repo carrying all of the dependencies for the ODE solver's callback system. 3. It started inviting circular dependencies. Some of these problems are pervasive. We need to get the simplest form to GalacticOptim.jl, Quadrature.jl, NonlinearSolve.jl, etc. which are becoming the DiffEq's of their respective regimes (though there's not as much to do in those areas). But this meant NonlinearSolve.jl needed NonlinearProblem, but NonlinearSolve.jl was used in the callbacks, so... it had its own problem type. This gives an issue to a system like MTK which then has no unique type to target. 4. There's a lot of churn on the DiffEq solver portions of DiffEqBase, but relatively little on the actual interface portions. So it makes sense to let them be more stable. 5. There's a ton of packages which are @requires in DiffEqBase, which adds to the compile time. This was done because the library was supposed to avoid dependencies, but as the central piece of the differential equation solvers it really couldn't, and so it used required to fake lower dependencies but some of those like ForwardDiff.jl should really be promoted to being a real dependency now that it doesn't have the "idea" of being no-dependency. For backwards compatibility, DiffEqBase reexports SciMLBase and imports a ton of the abstract types it used to have. This should allow for the migration to take place over time, while still fixing some of the MTK and NonlinearSolve issues that need a fix ASAP. DiffEqBase will have a companion PR to match it. There are some remaining issues here though: 1. I would like to get rid of the LabelledArrays dependency. Is there a good generic to use for the names? `nameof`? 2. Is the split at the right level? 3. I want to drop the Queryverse stuff like TableTraits and IteratorInterfaceExtensions since we have Tables.jl

ChrisRackauckas · 2021-01-26T04:18:22Z

Tested SciML/NonlinearSolve.jl#25 works.

YingboMa

Could you move the some of the DiffEqBase tests here, too? It's probably pretty tricky to do, though.

ChrisRackauckas · 2021-01-26T05:00:52Z

I plan to add a bunch of downstream tests.

YingboMa · 2021-01-26T05:04:57Z

Yeah, makes sense. I like this a lot. This should cut down a lot of complication time. BTW, do we want to move OrdinaryDiffEq's generic dense output to DiffEqBase? Then other solvers and callbacks can use that interface.

devmotion · 2021-01-26T06:35:33Z

I agree, it's great to separate the more basic non-DiffEq stuff. Is the plan also to move common functionality (e.g., generic interpolation, initialization, and keyword arguments) that appears in OrdinaryDiffEq and StochasticDiffEq (and to some extent also in DelayDiffEq and StochasticDelayDiffEq) to DiffEqBase then? I assume this would make it a bit easier to synchronize changes in OrdinaryDiffEq, StochasticDiffEq, DelayDiffEq etc. and maybe would allow us to drop the dependency on OrdinaryDiffEq in StochasticDiffEq.

devmotion · 2021-01-26T06:44:49Z

Project.toml

+StaticArrays = "90137ffa-7385-5640-81b9-e52037218182"
+Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
+Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"
+TreeViews = "a2a6695c-b41b-5b7d-aed9-dbfdeacea5d7"


I was a bit surprised to see that the latest release is from July 2018 and the repo contains no Project.toml but only a REQUIRE file. But I assume it just indicates that the package is very stable and no changes are expected?

It's super stable. @pfitzseb

Yep. I don't think there are any actively maintained consumers of TreeViews's tree views anymore though ;)

What do we define for VSCode and Juno nowadays then? TreeViews still seems to work for Juno?

Yes. VSCode's tree views (in the workspace) aren't customizable atm.

devmotion

Just noticed that we defined eltype for instances of DESolutionRow and AbstractSciMLOperator although the Julia docs recommend to define it for types.

src/operators.jl

src/tabletraits.jl

devmotion · 2021-01-26T07:10:23Z

There's a ton of packages which are @requires in DiffEqBase, which adds to the compile time.

Did you check if the PRs improve the compile time of DiffEqBase?

I would like to get rid of the LabelledArrays dependency. Is there a good generic to use for the names? nameof?

It seems https://github.com/SciML/DiffEqBase.jl/blob/b5ac95fedb065af3b9aeb95a09eb2fa043b7424f/src/solutions/solution_interface.jl#L158 is equivalent to

    return keys(sol.u[1])

so maybe use keys?

ChrisRackauckas · 2021-01-26T13:58:54Z

Did you check if the PRs improve the compile time of DiffEqBase?

No, that's a secondary concern here. Right now it's a big shuffle so I'm more concerned about correctness.

ChrisRackauckas · 2021-01-26T14:04:33Z

Is the plan also to move common functionality (e.g., generic interpolation, initialization, and keyword arguments) that appears in OrdinaryDiffEq and StochasticDiffEq (and to some extent also in DelayDiffEq and StochasticDelayDiffEq) to DiffEqBase then? I assume this would make it a bit easier to synchronize changes in OrdinaryDiffEq, StochasticDiffEq, DelayDiffEq etc. and maybe would allow us to drop the dependency on OrdinaryDiffEq in StochasticDiffEq.

Yes, precisely. And we'll want to PR to upstream packages telling them about SciMLBase.jl.

BTW, do we want to move OrdinaryDiffEq's generic dense output to DiffEqBase?

Oh buddy, it's even worse than that:

https://github.com/SciML/DiffEqBase.jl/blob/master/src/interpolation.jl

That's the kind of thing I hope we can start cleaning up.

so maybe use keys?

Yes that looks like it works.

Co-authored-by: David Widmann <[email protected]>

quinnj · 2021-01-26T15:50:33Z

src/tabletraits.jl

@@ -0,0 +1,42 @@
+Tables.isrowtable(::Type{<:DESolution}) = true
+Tables.columns(x::DESolution) = Tables.columntable(Tables.rows(x))


Did we have this definition previously? At first glance, it doesn't seem necessary (the fallback in Tables.jl should do the same thing)

Otherwise, all the Tables.jl stuff here looks great

ChrisRackauckas · 2021-01-27T00:25:22Z

Churning through those downstream tests 😃

ChrisRackauckas · 2021-01-27T00:48:59Z

DelayDiffEq passed.

ChrisRackauckas · 2021-01-27T00:51:33Z

LSODA passed.

ChrisRackauckas · 2021-01-27T00:52:45Z

Sundials passed.

ChrisRackauckas · 2021-01-27T01:08:07Z

DiffEqCallbacks passed.

ChrisRackauckas · 2021-01-27T01:08:18Z

DiffEqJump passed

ChrisRackauckas · 2021-01-27T01:38:01Z

DiffEqBase and ModelingToolkit passed

ChrisRackauckas · 2021-01-27T01:38:25Z

DiffEqSensitivity passed

ChrisRackauckas · 2021-01-27T01:39:57Z

StochasticDelayDiffEq passed

ChrisRackauckas · 2021-01-27T01:57:54Z

DiffEqUncertainty passed

This is kind of a hack, but it looks like there was a major regression in that library and I'm not quite sure why, but given I found this while working on SciML/SciMLBase.jl#1 I'm putting a fix to ElasticArrays right inline and will upstream that later. I'm kind of confused because I don't see a recent tag from ElasticArrays but it could've been a versioning thing.

Rebase

Update alg_traits.jl

ChrisRackauckas added 2 commits January 25, 2021 22:45

Remove non-Tables.jl

55fb554

ChrisRackauckas requested review from devmotion, kanav99 and YingboMa January 26, 2021 03:46

ChrisRackauckas mentioned this pull request Jan 26, 2021

Split SciMLBase from DiffEqBase SciML/DiffEqBase.jl#638

Merged

ChrisRackauckas added 3 commits January 25, 2021 22:49

fix compats

589672b

inits and solves

995953f

NonlinearSolve passes

3e52bc2

YingboMa reviewed Jan 26, 2021

View reviewed changes

devmotion reviewed Jan 26, 2021

View reviewed changes

src/operators.jl Outdated Show resolved Hide resolved

src/tabletraits.jl Outdated Show resolved Hide resolved

ChrisRackauckas and others added 5 commits January 26, 2021 09:05

move diffeq-specific utils

ad0b05f

remove labelledarrays

385d37b

don't forget remake!

381ae09

Update src/operators.jl

b8a9723

Co-authored-by: David Widmann <[email protected]>

Update src/tabletraits.jl

2b1a3a0

Co-authored-by: David Widmann <[email protected]>

quinnj reviewed Jan 26, 2021

View reviewed changes

ChrisRackauckas added 3 commits January 26, 2021 15:53

more minor changes to make OrdinaryDiffEq work

7659510

Merge remote-tracking branch 'origin/split' into split

10c21fc

tons of updates and downstream tests

70779ac

many more tests passing

2c8c059

ChrisRackauckas added 2 commits January 26, 2021 21:28

revert operator change

ff2435e

don't reexport resize!

f2ad318

ChrisRackauckas mentioned this pull request Jan 27, 2021

Fix ElasticArray tests SciML/OrdinaryDiffEq.jl#1337

Merged

ChrisRackauckas merged commit 7151bbe into master Jan 27, 2021

ChrisRackauckas deleted the split branch January 27, 2021 03:25

ChrisRackauckas pushed a commit that referenced this pull request Mar 30, 2021

Merge pull request #1 from SciML/master

a201053

Rebase

Vaibhavdixit02 pushed a commit that referenced this pull request Mar 11, 2024

Merge pull request #1 from ParasPuneetSingh/ParasPuneetSingh-patch-1

3bb183c

Update alg_traits.jl

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Split SciMLBase from DiffEqBase #1

Split SciMLBase from DiffEqBase #1

ChrisRackauckas commented Jan 26, 2021 •

edited

Loading

ChrisRackauckas commented Jan 26, 2021

YingboMa left a comment

ChrisRackauckas commented Jan 26, 2021

YingboMa commented Jan 26, 2021

devmotion commented Jan 26, 2021 •

edited

Loading

devmotion Jan 26, 2021

ChrisRackauckas Jan 26, 2021 •

edited

Loading

pfitzseb Jan 26, 2021

ChrisRackauckas Jan 26, 2021

pfitzseb Jan 26, 2021

devmotion left a comment

devmotion commented Jan 26, 2021 •

edited

Loading

ChrisRackauckas commented Jan 26, 2021

ChrisRackauckas commented Jan 26, 2021

quinnj Jan 26, 2021

quinnj Jan 26, 2021

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 27, 2021

		@@ -0,0 +1,42 @@
		Tables.isrowtable(::Type{<:DESolution}) = true
		Tables.columns(x::DESolution) = Tables.columntable(Tables.rows(x))

Split SciMLBase from DiffEqBase #1

Split SciMLBase from DiffEqBase #1

Conversation

ChrisRackauckas commented Jan 26, 2021 • edited Loading

ChrisRackauckas commented Jan 26, 2021

YingboMa left a comment

Choose a reason for hiding this comment

ChrisRackauckas commented Jan 26, 2021

YingboMa commented Jan 26, 2021

devmotion commented Jan 26, 2021 • edited Loading

devmotion Jan 26, 2021

Choose a reason for hiding this comment

ChrisRackauckas Jan 26, 2021 • edited Loading

Choose a reason for hiding this comment

pfitzseb Jan 26, 2021

Choose a reason for hiding this comment

ChrisRackauckas Jan 26, 2021

Choose a reason for hiding this comment

pfitzseb Jan 26, 2021

Choose a reason for hiding this comment

devmotion left a comment

Choose a reason for hiding this comment

devmotion commented Jan 26, 2021 • edited Loading

ChrisRackauckas commented Jan 26, 2021

ChrisRackauckas commented Jan 26, 2021

quinnj Jan 26, 2021

Choose a reason for hiding this comment

quinnj Jan 26, 2021

Choose a reason for hiding this comment

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 27, 2021

ChrisRackauckas commented Jan 26, 2021 •

edited

Loading

devmotion commented Jan 26, 2021 •

edited

Loading

ChrisRackauckas Jan 26, 2021 •

edited

Loading

devmotion commented Jan 26, 2021 •

edited

Loading