Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Where is JuMP's sparse AD code, and can DifferentiationInterface help? #3806

Closed
gdalle opened this issue Aug 14, 2024 · 5 comments
Closed

Where is JuMP's sparse AD code, and can DifferentiationInterface help? #3806

gdalle opened this issue Aug 14, 2024 · 5 comments

Comments

@gdalle
Copy link
Contributor

gdalle commented Aug 14, 2024

Hi there @odow and friends!

Looking at our work on sparse autodiff with @adrhill and @amontoison, I've been wondering how it could be useful to JuMP. We have developed a combination of three new packages:

From what I understand, JuMP's current sparse AD engine is Nonlinear.ReverseAD, but there are also experiments going on in MathOptSymbolicAD.jl. Did I miss anything?

I'm not suggesting that DifferentiationInterface and friends should replace your own default AD solution. But since you say in the docs that testing other AD backends would be nice, maybe there's an angle there?
In any case, perhaps DI could be a nice addition to the docs page on autodiff of user-defined operators?

@odow
Copy link
Member

odow commented Aug 14, 2024

Did I miss anything?

Nope. That is the situation. All nonlinear/AD stuff is in the MOI.Nonllinear module. There is nothing in JuMP.

But since you say in the docs that testing other AD backends would be nice, maybe there's an angle there?

New AD backends need to be implemented as an oracle like this:

https://github.com/lanl-ansi/MathOptSymbolicAD.jl/blob/e5d3eaa1f56346d53a5ef8f895743ab3e89ff38c/src/nonlinear_oracle.jl#L278-L496

In any case, perhaps DI could be a nice addition to the docs page on autodiff of user-defined operators?

Feel free to open a PR 😄

@amontoison
Copy link
Contributor

amontoison commented Aug 15, 2024

What could be useful for JuMP is star coloring for the Hessian of the Lagrangian.
It requires more colors compared to acyclic coloring (the coloring used by default in MOI.Nonlinear), but it puts less pressure on memory by allowing not storing all columns of the compressed Hessian for very large problems.

The issue is that the code related to nonlinear stuff in MOI is highly specialized for what they use in JuMP, so it's probably not easy to create a bridge.

@odow
Copy link
Member

odow commented Aug 18, 2024

I have no plans to work on a DI interface, but anyone interested is encouraged to give it a go. It can (and should) be done first in a separate repo like MathOptSymbolicAD.jl. One goal of refactoring the nonlinear stuff out of JuMP was so that we could experiment without needing to make source code changes to JuMP or MathOptInterface.

For the user-defined gradients with DI, I'm open to including this in the documentation, but it is lower priority, so perhaps we could close this issue in favor of adding a TODO item to #2348.

@amontoison
Copy link
Contributor

I agree with that Oscar.

@odow
Copy link
Member

odow commented Aug 19, 2024

Closing in favor of #2348 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

3 participants