diff --git a/DifferentiationInterface/dev/.documenter-siteinfo.json b/DifferentiationInterface/dev/.documenter-siteinfo.json index 5826491f5..c4c1a9e4a 100644 --- a/DifferentiationInterface/dev/.documenter-siteinfo.json +++ b/DifferentiationInterface/dev/.documenter-siteinfo.json @@ -1 +1 @@ -{"documenter":{"julia_version":"1.10.4","generation_timestamp":"2024-07-19T08:29:14","documenter_version":"1.5.0"}} \ No newline at end of file +{"documenter":{"julia_version":"1.10.4","generation_timestamp":"2024-07-22T18:24:19","documenter_version":"1.5.0"}} \ No newline at end of file diff --git a/DifferentiationInterface/dev/api/index.html b/DifferentiationInterface/dev/api/index.html index 88b05bdac..83037f4ad 100644 --- a/DifferentiationInterface/dev/api/index.html +++ b/DifferentiationInterface/dev/api/index.html @@ -1,27 +1,27 @@ -API · DifferentiationInterface.jl
GitHub

API

DifferentiationInterface — Module
DifferentiationInterface

An interface to various automatic differentiation backends in Julia.

Exports

source

First order

Pushforward

DifferentiationInterface.prepare_pushforward — Function
prepare_pushforward(f,     backend, x, dx) -> extras
-prepare_pushforward(f!, y, backend, x, dx) -> extras

Create an extras object that can be given to pushforward and its variants.

Warning

If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.

source
DifferentiationInterface.prepare_pushforward_same_point — Function
prepare_pushforward_same_point(f,     backend, x, dx) -> extras_same
-prepare_pushforward_same_point(f!, y, backend, x, dx) -> extras_same

Create an extras_same object that can be given to pushforward and its variants if they are applied at the same point x.

Warning

If the function or the point changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.

source
DifferentiationInterface.pushforward! — Function
pushforward!(f,     dy, backend, x, dx, [extras]) -> dy
-pushforward!(f!, y, dy, backend, x, dx, [extras]) -> dy

Compute the pushforward of the function f at point x with seed dx, overwriting dy.

To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.

Tip

Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named jvp!.

source
DifferentiationInterface.value_and_pushforward — Function
value_and_pushforward(f,     backend, x, dx, [extras]) -> (y, dy)
-value_and_pushforward(f!, y, backend, x, dx, [extras]) -> (y, dy)

Compute the value and the pushforward of the function f at point x with seed dx.

To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.

Tip

Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named value_and_jvp.

Info

Required primitive for forward mode backends.

source
DifferentiationInterface.value_and_pushforward! — Function
value_and_pushforward!(f,     dy, backend, x, dx, [extras]) -> (y, dy)
-value_and_pushforward!(f!, y, dy, backend, x, dx, [extras]) -> (y, dy)

Compute the value and the pushforward of the function f at point x with seed dx, overwriting dy.

To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.

Tip

Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named value_and_jvp!.

source

Pullback

DifferentiationInterface.prepare_pullback — Function
prepare_pullback(f,     backend, x, dy) -> extras
-prepare_pullback(f!, y, backend, x, dy) -> extras

Create an extras object that can be given to pullback and its variants.

Warning

If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.

source
DifferentiationInterface.prepare_pullback_same_point — Function
prepare_pullback_same_point(f,     backend, x, dy) -> extras_same
-prepare_pullback_same_point(f!, y, backend, x, dy) -> extras_same

Create an extras_same object that can be given to pullback and its variants if they are applied at the same point x.

Warning

If the function or the point changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.

source
DifferentiationInterface.pullback — Function
pullback(f,     backend, x, dy, [extras]) -> dx
-pullback(f!, y, backend, x, dy, [extras]) -> dx

Compute the pullback of the function f at point x with seed dy.

To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.

Tip

Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named vjp.

source
DifferentiationInterface.pullback! — Function
pullback!(f,     dx, backend, x, dy, [extras]) -> dx
-pullback!(f!, y, dx, backend, x, dy, [extras]) -> dx

Compute the pullback of the function f at point x with seed dy, overwriting dx.

To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.

Tip

Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named vjp!.

source
DifferentiationInterface.value_and_pullback — Function
value_and_pullback(f,     backend, x, dy, [extras]) -> (y, dx)
-value_and_pullback(f!, y, backend, x, dy, [extras]) -> (y, dx)

Compute the value and the pullback of the function f at point x with seed dy.

To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.

Tip

Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named value_and_vjp.

Info

Required primitive for reverse mode backends.

source
DifferentiationInterface.value_and_pullback! — Function
value_and_pullback!(f,     dx, backend, x, dy, [extras]) -> (y, dx)
-value_and_pullback!(f!, y, dx, backend, x, dy, [extras]) -> (y, dx)

Compute the value and the pullback of the function f at point x with seed dy, overwriting dx.

To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.

Tip

Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named value_and_vjp!.

source

Derivative

DifferentiationInterface.prepare_derivative — Function
prepare_derivative(f,     backend, x) -> extras
-prepare_derivative(f!, y, backend, x) -> extras

Create an extras object that can be given to derivative and its variants.

Warning

If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.

source
DifferentiationInterface.derivative! — Function
derivative!(f,     der, backend, x, [extras]) -> der
-derivative!(f!, y, der, backend, x, [extras]) -> der

Compute the derivative of the function f at point x, overwriting der.

To improve performance via operator preparation, refer to prepare_derivative.

source
DifferentiationInterface.value_and_derivative! — Function
value_and_derivative!(f,     der, backend, x, [extras]) -> (y, der)
-value_and_derivative!(f!, y, der, backend, x, [extras]) -> (y, der)

Compute the value and the derivative of the function f at point x, overwriting der.

To improve performance via operator preparation, refer to prepare_derivative.

source

Gradient

DifferentiationInterface.prepare_gradient — Function
prepare_gradient(f, backend, x) -> extras

Create an extras object that can be given to gradient and its variants.

Warning

If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.

source

Jacobian

DifferentiationInterface.prepare_jacobian — Function
prepare_jacobian(f,     backend, x) -> extras
-prepare_jacobian(f!, y, backend, x) -> extras

Create an extras object that can be given to jacobian and its variants.

Warning

If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.

source
DifferentiationInterface.jacobian! — Function
jacobian!(f,     jac, backend, x, [extras]) -> jac
-jacobian!(f!, y, jac, backend, x, [extras]) -> jac

Compute the Jacobian matrix of the function f at point x, overwriting jac.

To improve performance via operator preparation, refer to prepare_jacobian.

source
DifferentiationInterface.value_and_jacobian! — Function
value_and_jacobian!(f,     jac, backend, x, [extras]) -> (y, jac)
-value_and_jacobian!(f!, y, jac, backend, x, [extras]) -> (y, jac)

Compute the value and the Jacobian matrix of the function f at point x, overwriting jac.

To improve performance via operator preparation, refer to prepare_jacobian.

source

Second order

DifferentiationInterface.SecondOrder — Type
SecondOrder

Combination of two backends for second-order differentiation.

Danger

SecondOrder backends do not support first-order operators.

Constructor

SecondOrder(outer_backend, inner_backend)

Fields

  • outer::ADTypes.AbstractADType: backend for the outer differentiation

  • inner::ADTypes.AbstractADType: backend for the inner differentiation

source

Second derivative

Hessian-vector product

DifferentiationInterface.prepare_hvp — Function
prepare_hvp(f, backend, x, dx) -> extras

Create an extras object that can be given to hvp and its variants.

Warning

If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.

source
DifferentiationInterface.prepare_hvp_same_point — Function
prepare_hvp_same_point(f, backend, x, dx) -> extras_same

Create an extras_same object that can be given to hvp and its variants if they are applied at the same point x.

Warning

If the function or the point changes in any way, the result of preparation will be invalidated, and you will need to run it again.

source

Hessian

DifferentiationInterface.prepare_hessian — Function
prepare_hessian(f, backend, x) -> extras

Create an extras object that can be given to hessian and its variants.

Warning

If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.

source

Utilities

Backend queries

DifferentiationInterface.check_hessian — Function
check_hessian(backend)

Check whether backend supports second order differentiation by trying to compute a hessian.

Warning

Might take a while due to compilation time.

source

Backend switch

DifferentiationInterface.DifferentiateWith — Type
DifferentiateWith

Callable function wrapper that enforces differentiation with a specified (inner) backend.

This works by defining new rules overriding the behavior of the outer backend that would normally be used.

Warning

This is an experimental functionality, whose API cannot yet be considered stable. At the moment, it only supports one-argument functions, and rules are only defined for ChainRules.jl-compatible outer backends.

Fields

  • f: the function in question
  • backend::AbstractADType: the inner backend to use for differentiation

Constructor

DifferentiateWith(f, backend)

Example

using DifferentiationInterface
+API · DifferentiationInterface.jl
GitHub
API
DifferentiationInterface — Module
DifferentiationInterface
An interface to various automatic differentiation backends in Julia.
Exports
source
First order
Pushforward
DifferentiationInterface.prepare_pushforward — Function
prepare_pushforward(f,     backend, x, dx) -> extras
+prepare_pushforward(f!, y, backend, x, dx) -> extras
Create an extras object that can be given to pushforward and its variants.
Warning
If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.
source
DifferentiationInterface.prepare_pushforward_same_point — Function
prepare_pushforward_same_point(f,     backend, x, dx) -> extras_same
+prepare_pushforward_same_point(f!, y, backend, x, dx) -> extras_same
Create an extras_same object that can be given to pushforward and its variants if they are applied at the same point x.
Warning
If the function or the point changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.
source
DifferentiationInterface.pushforward! — Function
pushforward!(f,     dy, backend, x, dx, [extras]) -> dy
+pushforward!(f!, y, dy, backend, x, dx, [extras]) -> dy
Compute the pushforward of the function f at point x with seed dx, overwriting dy.
To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.
Tip
Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named jvp!.
source
DifferentiationInterface.value_and_pushforward — Function
value_and_pushforward(f,     backend, x, dx, [extras]) -> (y, dy)
+value_and_pushforward(f!, y, backend, x, dx, [extras]) -> (y, dy)
Compute the value and the pushforward of the function f at point x with seed dx.
To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.
Tip
Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named value_and_jvp.
Info
Required primitive for forward mode backends.
source
DifferentiationInterface.value_and_pushforward! — Function
value_and_pushforward!(f,     dy, backend, x, dx, [extras]) -> (y, dy)
+value_and_pushforward!(f!, y, dy, backend, x, dx, [extras]) -> (y, dy)
Compute the value and the pushforward of the function f at point x with seed dx, overwriting dy.
To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.
Tip
Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named value_and_jvp!.
source
Pullback
DifferentiationInterface.prepare_pullback — Function
prepare_pullback(f,     backend, x, dy) -> extras
+prepare_pullback(f!, y, backend, x, dy) -> extras
Create an extras object that can be given to pullback and its variants.
Warning
If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.
source
DifferentiationInterface.prepare_pullback_same_point — Function
prepare_pullback_same_point(f,     backend, x, dy) -> extras_same
+prepare_pullback_same_point(f!, y, backend, x, dy) -> extras_same
Create an extras_same object that can be given to pullback and its variants if they are applied at the same point x.
Warning
If the function or the point changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.
source
DifferentiationInterface.pullback — Function
pullback(f,     backend, x, dy, [extras]) -> dx
+pullback(f!, y, backend, x, dy, [extras]) -> dx
Compute the pullback of the function f at point x with seed dy.
To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.
Tip
Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named vjp.
source
DifferentiationInterface.pullback! — Function
pullback!(f,     dx, backend, x, dy, [extras]) -> dx
+pullback!(f!, y, dx, backend, x, dy, [extras]) -> dx
Compute the pullback of the function f at point x with seed dy, overwriting dx.
To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.
Tip
Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named vjp!.
source
DifferentiationInterface.value_and_pullback — Function
value_and_pullback(f,     backend, x, dy, [extras]) -> (y, dx)
+value_and_pullback(f!, y, backend, x, dy, [extras]) -> (y, dx)
Compute the value and the pullback of the function f at point x with seed dy.
To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.
Tip
Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named value_and_vjp.
Info
Required primitive for reverse mode backends.
source
DifferentiationInterface.value_and_pullback! — Function
value_and_pullback!(f,     dx, backend, x, dy, [extras]) -> (y, dx)
+value_and_pullback!(f!, y, dx, backend, x, dy, [extras]) -> (y, dx)
Compute the value and the pullback of the function f at point x with seed dy, overwriting dx.
To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.
Tip
Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named value_and_vjp!.
source
Derivative
DifferentiationInterface.prepare_derivative — Function
prepare_derivative(f,     backend, x) -> extras
+prepare_derivative(f!, y, backend, x) -> extras
Create an extras object that can be given to derivative and its variants.
Warning
If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.
source
DifferentiationInterface.derivative! — Function
derivative!(f,     der, backend, x, [extras]) -> der
+derivative!(f!, y, der, backend, x, [extras]) -> der
Compute the derivative of the function f at point x, overwriting der.
To improve performance via operator preparation, refer to prepare_derivative.
source
DifferentiationInterface.value_and_derivative! — Function
value_and_derivative!(f,     der, backend, x, [extras]) -> (y, der)
+value_and_derivative!(f!, y, der, backend, x, [extras]) -> (y, der)
Compute the value and the derivative of the function f at point x, overwriting der.
To improve performance via operator preparation, refer to prepare_derivative.
source
Gradient
DifferentiationInterface.prepare_gradient — Function
prepare_gradient(f, backend, x) -> extras
Create an extras object that can be given to gradient and its variants.
Warning
If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.
source
Jacobian
DifferentiationInterface.prepare_jacobian — Function
prepare_jacobian(f,     backend, x) -> extras
+prepare_jacobian(f!, y, backend, x) -> extras
Create an extras object that can be given to jacobian and its variants.
Warning
If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.
source
DifferentiationInterface.jacobian! — Function
jacobian!(f,     jac, backend, x, [extras]) -> jac
+jacobian!(f!, y, jac, backend, x, [extras]) -> jac
Compute the Jacobian matrix of the function f at point x, overwriting jac.
To improve performance via operator preparation, refer to prepare_jacobian.
source
DifferentiationInterface.value_and_jacobian! — Function
value_and_jacobian!(f,     jac, backend, x, [extras]) -> (y, jac)
+value_and_jacobian!(f!, y, jac, backend, x, [extras]) -> (y, jac)
Compute the value and the Jacobian matrix of the function f at point x, overwriting jac.
To improve performance via operator preparation, refer to prepare_jacobian.
source
Second order
DifferentiationInterface.SecondOrder — Type
SecondOrder
Combination of two backends for second-order differentiation.
Danger
SecondOrder backends do not support first-order operators.
Constructor
SecondOrder(outer_backend, inner_backend)
Fields
  • outer::ADTypes.AbstractADType: backend for the outer differentiation
  • inner::ADTypes.AbstractADType: backend for the inner differentiation
source
Second derivative
Hessian-vector product
DifferentiationInterface.prepare_hvp — Function
prepare_hvp(f, backend, x, dx) -> extras
Create an extras object that can be given to hvp and its variants.
Warning
If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.
source
DifferentiationInterface.prepare_hvp_same_point — Function
prepare_hvp_same_point(f, backend, x, dx) -> extras_same
Create an extras_same object that can be given to hvp and its variants if they are applied at the same point x.
Warning
If the function or the point changes in any way, the result of preparation will be invalidated, and you will need to run it again.
source
Hessian
DifferentiationInterface.prepare_hessian — Function
prepare_hessian(f, backend, x) -> extras
Create an extras object that can be given to hessian and its variants.
Warning
If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.
source
Utilities
Backend queries
DifferentiationInterface.check_hessian — Function
check_hessian(backend)
Check whether backend supports second order differentiation by trying to compute a hessian.
Warning
Might take a while due to compilation time.
source
Backend switch
DifferentiationInterface.DifferentiateWith — Type
DifferentiateWith
Callable function wrapper that enforces differentiation with a specified (inner) backend.
This works by defining new rules overriding the behavior of the outer backend that would normally be used.
Warning
This is an experimental functionality, whose API cannot yet be considered stable. At the moment, it only supports one-argument functions, and rules are only defined for ChainRules.jl-compatible outer backends.
Fields
  • f: the function in question
  • backend::AbstractADType: the inner backend to use for differentiation
Constructor
DifferentiateWith(f, backend)
Example
using DifferentiationInterface
 import ForwardDiff, Zygote
 
 function f(x)
@@ -37,7 +37,7 @@
 # output
 
 1-element Vector{Float64}:
- 1.0
source
Sparsity detection
DifferentiationInterface.DenseSparsityDetector — Type
DenseSparsityDetector
Sparsity pattern detector satisfying the detection API of ADTypes.jl.
The nonzeros in a Jacobian or Hessian are detected by computing the relevant matrix with dense AD, and thresholding the entries with a given tolerance (which can be numerically inaccurate).
Warning
This detector can be very slow, and should only be used if its output can be exploited multiple times to compute many sparse matrices. 
Danger
In general, the sparsity pattern you obtain can depend on the provided input x. If you want to reuse the pattern, make sure that it is input-agnostic.
Fields
  • backend::AbstractADType is the dense AD backend used under the hood
  • atol::Float64 is the minimum magnitude of a matrix entry to be considered nonzero
Constructor
DenseSparsityDetector(backend; atol, method=:iterative)
The keyword argument method::Symbol can be either:
  • :iterative: compute the matrix in a sequence of matrix-vector products (memory-efficient)
  • :direct: compute the matrix all at once (memory-hungry but sometimes faster).
Note that the constructor is type-unstable because method ends up being a type parameter of the DenseSparsityDetector object (this is not part of the API and might change).
Examples
using ADTypes, DifferentiationInterface, SparseArrays
+ 1.0
source
Sparsity detection
DifferentiationInterface.DenseSparsityDetector — Type
DenseSparsityDetector
Sparsity pattern detector satisfying the detection API of ADTypes.jl.
The nonzeros in a Jacobian or Hessian are detected by computing the relevant matrix with dense AD, and thresholding the entries with a given tolerance (which can be numerically inaccurate).
Warning
This detector can be very slow, and should only be used if its output can be exploited multiple times to compute many sparse matrices. 
Danger
In general, the sparsity pattern you obtain can depend on the provided input x. If you want to reuse the pattern, make sure that it is input-agnostic.
Fields
  • backend::AbstractADType is the dense AD backend used under the hood
  • atol::Float64 is the minimum magnitude of a matrix entry to be considered nonzero
Constructor
DenseSparsityDetector(backend; atol, method=:iterative)
The keyword argument method::Symbol can be either:
  • :iterative: compute the matrix in a sequence of matrix-vector products (memory-efficient)
  • :direct: compute the matrix all at once (memory-hungry but sometimes faster).
Note that the constructor is type-unstable because method ends up being a type parameter of the DenseSparsityDetector object (this is not part of the API and might change).
Examples
using ADTypes, DifferentiationInterface, SparseArrays
 import ForwardDiff
 
 detector = DenseSparsityDetector(AutoForwardDiff(); atol=1e-5, method=:direct)
@@ -60,9 +60,9 @@
 # output
 
 1×2 SparseMatrixCSC{Bool, Int64} with 1 stored entry:
- 1  ⋅
source
Internals
The following is not part of the public API.
DifferentiationInterface.Batch — Type
Batch{B,T}
Efficient storage for B elements of type T (NTuple wrapper).
A Batch can be used as seed to trigger batched-mode pushforward, pullback and hvp.
Fields
  • elements::NTuple{B,T}
source
ADTypes.mode — Method
mode(backend::SecondOrder)
Return the outer mode of the second-order backend.
source
DifferentiationInterface.basis — Method
basis(backend, a::AbstractArray, i::CartesianIndex)
Construct the i-th stardard basis array in the vector space of a with element type eltype(a).
Note
If an AD backend benefits from a more specialized basis array implementation, this function can be extended on the backend type.
source
DifferentiationInterface.nested — Method
nested(backend)
Return a possibly modified backend that can work while nested inside another differentiation procedure.
At the moment, this is only useful for Enzyme, which needs autodiff_deferred to be compatible with higher-order differentiation.
source
DifferentiationInterface.pick_batchsize — Method
pick_batchsize(backend::AbstractADType, dimension::Integer)
Pick a reasonable batch size for batched derivative evaluation with a given total dimension.
Returns 1 for backends which have not overloaded it.
source

+
diff --git a/DifferentiationInterface/dev/backends/index.html b/DifferentiationInterface/dev/backends/index.html index 3b15cfee8..c89b39839 100644 --- a/DifferentiationInterface/dev/backends/index.html +++ b/DifferentiationInterface/dev/backends/index.html @@ -5,4 +5,4 @@ startOnLoad: true, theme: "neutral" }); - + diff --git a/DifferentiationInterface/dev/dev_guide/index.html b/DifferentiationInterface/dev/dev_guide/index.html index 7d9f45c7e..ebd1e4607 100644 --- a/DifferentiationInterface/dev/dev_guide/index.html +++ b/DifferentiationInterface/dev/dev_guide/index.html @@ -4,4 +4,4 @@ startOnLoad: true, theme: "neutral" }); - + diff --git a/DifferentiationInterface/dev/index.html b/DifferentiationInterface/dev/index.html index d413dd35d..780d70afa 100644 --- a/DifferentiationInterface/dev/index.html +++ b/DifferentiationInterface/dev/index.html @@ -20,4 +20,4 @@ startOnLoad: true, theme: "neutral" }); - + diff --git a/DifferentiationInterface/dev/operators/index.html b/DifferentiationInterface/dev/operators/index.html index a0ec40d80..d7161be6c 100644 --- a/DifferentiationInterface/dev/operators/index.html +++ b/DifferentiationInterface/dev/operators/index.html @@ -5,4 +5,4 @@ startOnLoad: true, theme: "neutral" }); - + diff --git a/DifferentiationInterface/dev/overloads/index.html b/DifferentiationInterface/dev/overloads/index.html index 31ec413cc..7a1660d9e 100644 --- a/DifferentiationInterface/dev/overloads/index.html +++ b/DifferentiationInterface/dev/overloads/index.html @@ -1,54 +1,54 @@ Overloads · DifferentiationInterface.jl
GitHub

Overloads

DifferentiationInterface.jl provides a handful of operators like gradient or jacobian, each with several variants:

  • out-of-place or in-place behavior
  • with or without primal output value
  • support for one-argument functions y = f(x) or two-argument functions f!(y, x)

While it is possible to define every operator using just pushforward and pullback, some backends have more efficient implementations of high-level operators. When they are available, we always call these backend-specific overloads.

The following tables summarize all implemented overloads for each backend. The cells can have three values:

  • ❌: the operator is not overloaded because the backend does not support it
  • ✅: the operator is overloaded
  • NA: the operator does not exist
Tip

Check marks (✅) are clickable and link to the source code.

Diffractor (forward/reverse)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient❌❌❌❌
jacobian❌❌❌❌
hessian❌❌NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅❌✅❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient❌❌❌❌
jacobian❌❌❌❌
hessian❌❌NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅❌✅❌

Two-argument functions f!(y, x)

Backend doesn't support mutating functions.

Enzyme (forward)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian❌❌NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅✅✅✅
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian❌❌NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅✅✅✅

Two-argument functions f!(y, x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
jacobian❌❌❌❌
pullback❌❌❌❌
pushforward❌❌✅❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
jacobian❌❌❌❌
pullback❌❌❌❌
pushforward❌❌✅❌

Enzyme (reverse)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian❌❌❌❌
hessian❌❌NANA
hvp❌❌NANA
pullback✅✅✅✅
pushforward❌❌❌❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian❌❌❌❌
hessian❌❌NANA
hvp❌❌NANA
pullback✅✅✅✅
pushforward❌❌❌❌

Two-argument functions f!(y, x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
jacobian❌❌❌❌
pullback❌❌✅❌
pushforward❌❌❌❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
jacobian❌❌❌❌
pullback❌❌✅❌
pushforward❌❌❌❌

FastDifferentiation (symbolic)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp✅✅NANA
pullback✅✅✅✅
pushforward✅✅✅✅
+
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp✅✅NANA
pullback✅✅✅✅
pushforward✅✅✅✅

Two-argument functions f!(y, x)

-
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback✅✅✅✅
pushforward✅✅✅✅
+
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback✅✅✅✅
pushforward✅✅✅✅

FiniteDiff (forward)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅❌✅❌
+
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅❌✅❌

Two-argument functions f!(y, x)

-
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback❌❌❌❌
pushforward❌❌✅❌
+
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback❌❌❌❌
pushforward❌❌✅❌

FiniteDifferences (forward)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian❌❌NANA
hvp❌❌NANA
pullback✅❌✅❌
pushforward✅❌✅❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian❌❌NANA
hvp❌❌NANA
pullback✅❌✅❌
pushforward✅❌✅❌

Two-argument functions f!(y, x)

Backend doesn't support mutating functions.

ForwardDiff (forward)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅✅✅✅
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅✅✅✅

Two-argument functions f!(y, x)

-
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback❌❌❌❌
pushforward✅✅✅✅
+
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback❌❌❌❌
pushforward✅✅✅✅

PolyesterForwardDiff (forward)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅✅✅✅
+
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅✅✅✅

Two-argument functions f!(y, x)

-
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback❌❌❌❌
pushforward✅✅✅✅
+
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback❌❌❌❌
pushforward✅✅✅✅

ReverseDiff (reverse)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌✅✅
pushforward❌❌❌❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌✅✅
pushforward❌❌❌❌

Two-argument functions f!(y, x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
jacobian✅✅✅✅
pullback✅✅✅✅
pushforward❌❌❌❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
jacobian✅✅✅✅
pullback✅✅✅✅
pushforward❌❌❌❌

Symbolics (symbolic)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅✅✅✅
+
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅✅✅✅

Two-argument functions f!(y, x)

-
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback❌❌❌❌
pushforward✅✅✅✅
+
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback❌❌❌❌
pushforward✅✅✅✅

Tapir (reverse)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient❌❌❌❌
jacobian❌❌❌❌
hessian❌❌NANA
hvp❌❌NANA
pullback✅✅✅✅
pushforward❌❌❌❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient❌❌❌❌
jacobian❌❌❌❌
hessian❌❌NANA
hvp❌❌NANA
pullback✅✅✅✅
pushforward❌❌❌❌

Two-argument functions f!(y, x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
jacobian❌❌❌❌
pullback❌❌✅❌
pushforward❌❌❌❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
jacobian❌❌❌❌
pullback❌❌✅❌
pushforward❌❌❌❌

Tracker (reverse)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian❌❌❌❌
hessian❌❌NANA
hvp❌❌NANA
pullback✅❌✅❌
pushforward❌❌❌❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian❌❌❌❌
hessian❌❌NANA
hvp❌❌NANA
pullback✅❌✅❌
pushforward❌❌❌❌

Two-argument functions f!(y, x)

Backend doesn't support mutating functions.

Zygote (reverse)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp✅✅NANA
pullback✅❌✅❌
pushforward❌❌❌❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp✅✅NANA
pullback✅❌✅❌
pushforward❌❌❌❌

Two-argument functions f!(y, x)

Backend doesn't support mutating functions.

+ diff --git a/DifferentiationInterface/dev/tutorial1/index.html b/DifferentiationInterface/dev/tutorial1/index.html index 8a72030cf..c68e6ff2f 100644 --- a/DifferentiationInterface/dev/tutorial1/index.html +++ b/DifferentiationInterface/dev/tutorial1/index.html @@ -21,14 +21,14 @@ 8.0 10.0

Was that fast? BenchmarkTools.jl helps you answer that question.

using BenchmarkTools
 
-@benchmark gradient($f, $backend, $x)
BenchmarkTools.Trial: 10000 samples with 188 evaluations.
- Range (min … max):  543.410 ns …  1.790 ms  ┊ GC (min … max):  0.00% … 99.95%
- Time  (median):     567.707 ns              ┊ GC (median):     0.00%
- Time  (mean ± σ):   851.148 ns ± 17.908 μs  ┊ GC (mean ± σ):  23.75% ±  4.08%
+@benchmark gradient($f, $backend, $x)
BenchmarkTools.Trial: 10000 samples with 186 evaluations.
+ Range (min … max):  560.570 ns …  1.805 ms  ┊ GC (min … max):  0.00% … 99.95%
+ Time  (median):     582.930 ns              ┊ GC (median):     0.00%
+ Time  (mean ± σ):   876.706 ns ± 18.052 μs  ┊ GC (mean ± σ):  23.29% ±  3.97%
 
-  ▂▆██▆▄▁  ▁▁▁ ▁                              ▁▂▃▃▄▅▅▅▃▂       ▂
-  ████████▇█████▇▆▆▆▅▆▄▄▃▄▃▃▃▃▃▄▁▃▄▃▄▁▄▁▃▃▁▁▁▅██████████▇▆▇▇█▇ █
-  543 ns        Histogram: log(frequency) by time       969 ns <
+  ▄▆█▇▅▂▁                                     ▂▃▄▃▃▄▄▄▃▂▁      ▂
+  ████████▇█████▇▆▆▅▆▆▆▅▆▆▅▅▅▄▁▃▄▄▃▄▁▃▃▁▄▄▃▁▄▆███████████▇▇▆▇█ █
+  561 ns        Histogram: log(frequency) by time       982 ns <
 
  Memory estimate: 848 bytes, allocs estimate: 4.

Not bad, but you can do better.

Overwriting a gradient

Since you know how much space your gradient will occupy (the same as your input x), you can pre-allocate that memory and offer it to AD. Some backends get a speed boost from this trick.

grad = similar(x)
 gradient!(f, grad, backend, x)
@@ -38,15 +38,15 @@
   6.0
   8.0
  10.0

The bang indicates that one of the arguments of gradient! might be mutated. More precisely, our convention is that every positional argument between the function and the backend is mutated (and the extras too, see below).

@benchmark gradient!($f, _grad, $backend, $x) evals=1 setup=(_grad=similar($x))
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
- Range (min … max):  561.000 ns …  13.204 μs  ┊ GC (min … max): 0.00% … 0.00%
- Time  (median):     601.000 ns               ┊ GC (median):    0.00%
- Time  (mean ± σ):   645.769 ns ± 192.828 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%
+ Range (min … max):  571.000 ns …  11.041 μs  ┊ GC (min … max): 0.00% … 0.00%
+ Time  (median):     611.000 ns               ┊ GC (median):    0.00%
+ Time  (mean ± σ):   648.217 ns ± 138.388 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%
 
-    ▁▇ █▁                                                        
-  ▂▃██▄██▅▃▄▃▃▂▃▄▃▂▃▃▃▁▃▃▃▁▃▃▃▁▃▂▂▁▂▂▂▁▂▂▂▁▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂ ▃
-  561 ns           Histogram: frequency by time         1.02 μs <
+   ▄▇ ██▆▄▅▄ ▄▄▃ ▄▃▃▂▃▃ ▃▃▂▁▂▁ ▁▁   ▁         ▁                 ▂
+  ███▁██████▁███▁██████▁██████▁███▁██████▁█▇▆▇██▁██▇▁▇▇▃▇▆▆▁▇▆▆ █
+  571 ns        Histogram: log(frequency) by time       1.01 μs <
 
- Memory estimate: 752 bytes, allocs estimate: 3.

For some reason the in-place version is not much better than your first attempt. However, it makes fewer allocations, thanks to the gradient vector you provided. Don't worry, you can get even more performance.

Preparing for multiple gradients

Internally, ForwardDiff.jl creates some data structures to keep track of things. These objects can be reused between gradient computations, even on different input values. We abstract away the preparation step behind a backend-agnostic syntax:

extras = prepare_gradient(f, backend, randn(eltype(x), size(x)))
DifferentiationInterfaceForwardDiffExt.ForwardDiffGradientExtras{ForwardDiff.GradientConfig{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5, Vector{ForwardDiff.Dual{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5}}}}(ForwardDiff.GradientConfig{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5, Vector{ForwardDiff.Dual{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5}}}((Partials(1.0, 0.0, 0.0, 0.0, 0.0), Partials(0.0, 1.0, 0.0, 0.0, 0.0), Partials(0.0, 0.0, 1.0, 0.0, 0.0), Partials(0.0, 0.0, 0.0, 1.0, 0.0), Partials(0.0, 0.0, 0.0, 0.0, 1.0)), ForwardDiff.Dual{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5}[Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(0.0,0.0,0.0,0.0,0.0,0.0), Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(0.0,0.0,0.0,0.0,0.0,0.0), Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(0.0,0.0,0.0,0.0,0.0,0.0), Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(0.0,0.0,0.0,0.0,0.0,0.0), Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(0.0,0.0,0.0,0.0,0.0,0.0)]))

You don't need to know what this object is, you just need to pass it to the gradient operator. Note that preparation does not depend on the actual components of the vector x, just on its type and size. You can thus reuse the extras for different values of the input.

grad = similar(x)
+ Memory estimate: 752 bytes, allocs estimate: 3.

For some reason the in-place version is not much better than your first attempt. However, it makes fewer allocations, thanks to the gradient vector you provided. Don't worry, you can get even more performance.

Preparing for multiple gradients

Internally, ForwardDiff.jl creates some data structures to keep track of things. These objects can be reused between gradient computations, even on different input values. We abstract away the preparation step behind a backend-agnostic syntax:

extras = prepare_gradient(f, backend, randn(eltype(x), size(x)))
DifferentiationInterfaceForwardDiffExt.ForwardDiffGradientExtras{ForwardDiff.GradientConfig{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5, Vector{ForwardDiff.Dual{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5}}}}(ForwardDiff.GradientConfig{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5, Vector{ForwardDiff.Dual{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5}}}((Partials(1.0, 0.0, 0.0, 0.0, 0.0), Partials(0.0, 1.0, 0.0, 0.0, 0.0), Partials(0.0, 0.0, 1.0, 0.0, 0.0), Partials(0.0, 0.0, 0.0, 1.0, 0.0), Partials(0.0, 0.0, 0.0, 0.0, 1.0)), ForwardDiff.Dual{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5}[Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(1.0,1.0,0.0,0.0,0.0,0.0), Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(2.0,0.0,1.0,0.0,0.0,0.0), Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(3.0,0.0,0.0,1.0,0.0,0.0), Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(4.0,0.0,0.0,0.0,1.0,0.0), Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(5.0,0.0,0.0,0.0,0.0,1.0)]))

You don't need to know what this object is, you just need to pass it to the gradient operator. Note that preparation does not depend on the actual components of the vector x, just on its type and size. You can thus reuse the extras for different values of the input.

grad = similar(x)
 gradient!(f, grad, backend, x, extras)
 grad  # has been mutated
5-element Vector{Float64}:
   2.0
@@ -57,13 +57,13 @@
     _grad=similar($x);
     _extras=prepare_gradient($f, $backend, $x)
 )
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
- Range (min … max):  60.000 ns …  1.332 μs  ┊ GC (min … max): 0.00% … 0.00%
- Time  (median):     70.000 ns              ┊ GC (median):    0.00%
- Time  (mean ± σ):   77.602 ns ± 30.797 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%
+ Range (min … max):  69.000 ns …  17.933 μs  ┊ GC (min … max): 0.00% … 0.00%
+ Time  (median):     80.000 ns               ┊ GC (median):    0.00%
+ Time  (mean ± σ):   87.783 ns ± 204.286 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%
 
-  ▅   █▆   ▅▂   ▃                             ▂    ▂    ▂     ▁
-  █▁▁▁██▁▁▁██▁▁▁██▁▁▁█▆▁▁▁▇▅▁▁▁▅▁▁▁▁▅▁▁▁▁█▁▁▁▁█▁▁▁▁█▁▁▁▁█▁▁▁▆ █
-  60 ns        Histogram: log(frequency) by time       180 ns <
+  ▇    █    ▄    ▁                       ▁    ▂▁   ▂           ▁
+  █▁▁▁▁█▁▁▁▆█▁▁▁▁█▁▁▁▁▇▁▁▁▁▅▁▁▁▄▄▁▁▁▇▅▁▁▁██▁▁▁██▁▁▁██▁▁▁██▁▁▁█ █
+  69 ns         Histogram: log(frequency) by time       191 ns <
 
  Memory estimate: 0 bytes, allocs estimate: 0.

Beware that the extras object is nearly always mutated by differentiation operators, even though it is given as the last positional argument.

Switching backends

The whole point of DifferentiationInterface.jl is that you can easily experiment with different AD solutions. Typically, for gradients, reverse mode AD might be a better fit, so let's try the state-of-the-art Enzyme.jl!

import Enzyme
 
@@ -76,17 +76,17 @@
     _grad=similar($x);
     _extras=prepare_gradient($f, $backend2, $x)
 )
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
- Range (min … max):  60.000 ns …  1.162 μs  ┊ GC (min … max): 0.00% … 0.00%
- Time  (median):     70.000 ns              ┊ GC (median):    0.00%
- Time  (mean ± σ):   66.991 ns ± 15.639 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%
+ Range (min … max):  60.000 ns … 371.000 ns  ┊ GC (min … max): 0.00% … 0.00%
+ Time  (median):     60.000 ns               ┊ GC (median):    0.00%
+ Time  (mean ± σ):   62.787 ns ±   9.020 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%
 
-  █▄        ▃█▅          ▃           ▁                        ▂
-  ██▁▁▁▁▁▁▁▁███▁▁▁▁▁▁▁▁▁▁█▇▁▁▁▁▁▁▁▁▁▄█▆▁▁▁▁▁▁▁▁▁▁▄▅▁▁▁▁▁▁▁▁▁▄ █
-  60 ns        Histogram: log(frequency) by time       110 ns <
+  █▅               ▁ ▅ ▃                 ▁                     ▁
+  ██▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁█▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▆▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▃▁█ █
+  60 ns         Histogram: log(frequency) by time        90 ns <
 
  Memory estimate: 0 bytes, allocs estimate: 0.

In short, DifferentiationInterface.jl allows for easy testing and comparison of AD backends. If you want to go further, check out the documentation of DifferentiationInterfaceTest.jl. This related package provides benchmarking utilities to compare backends and help you select the one that is best suited for your problem.

+ diff --git a/DifferentiationInterface/dev/tutorial2/index.html b/DifferentiationInterface/dev/tutorial2/index.html index e01d948bd..df8ba521b 100644 --- a/DifferentiationInterface/dev/tutorial2/index.html +++ b/DifferentiationInterface/dev/tutorial2/index.html @@ -57,27 +57,27 @@ -144.0 576.0 -432.0 ⋅ ⋅ ⋅ -1152.0 2560.0 -1440.0 160.0 -320.0 ⋅ ⋅ ⋅ ⋅ ⋅ -1440.0 1728.0

Sparse preparation

In the examples above, we didn't use preparation. Sparse preparation is more costly than dense preparation, but it is even more essential. Indeed, once preparation is done, sparse differentiation is much faster than dense differentiation, because it makes fewer calls to the underlying function. The speedup becomes very visible in large dimensions.

n = 1000
 jac_extras_dense = prepare_jacobian(f_sparse_vector, dense_first_order_backend, randn(n));
-jac_extras_sparse = prepare_jacobian(f_sparse_vector, sparse_first_order_backend, randn(n));
@benchmark jacobian($f_sparse_vector, $dense_first_order_backend, $(randn(n)), $jac_extras_dense) evals=1
BenchmarkTools.Trial: 688 samples with 1 evaluation.
- Range (min … max):  4.891 ms … 32.056 ms  ┊ GC (min … max):  0.00% … 50.45%
- Time  (median):     6.232 ms              ┊ GC (median):    17.38%
- Time  (mean ± σ):   6.228 ms ±  1.745 ms  ┊ GC (mean ± σ):  13.70% ±  8.60%
+jac_extras_sparse = prepare_jacobian(f_sparse_vector, sparse_first_order_backend, randn(n));
@benchmark jacobian($f_sparse_vector, $dense_first_order_backend, $(randn(n)), $jac_extras_dense) evals=1
BenchmarkTools.Trial: 677 samples with 1 evaluation.
+ Range (min … max):  4.996 ms … 37.733 ms  ┊ GC (min … max):  0.00% … 46.24%
+ Time  (median):     6.437 ms              ┊ GC (median):    19.08%
+ Time  (mean ± σ):   6.311 ms ±  1.814 ms  ┊ GC (mean ± σ):  15.00% ±  9.41%
 
-       ▁   ▃               ▁▄▅▃▃▂▁▄▄▄▇█▅▁                     
-  ▄▄██▇█▇▆▇███▄▅▃▁▂▁▁▁▂▄▄▆▇████████████████▄▃▂▃▂▁▂▂▂▁▃▂▂▂▁▃▂ ▄
-  4.89 ms        Histogram: frequency by time        7.58 ms <
+   ▂▄             ▂▅█▆▁▁                                      
+  ▄██▇▅▄▃▁▂▂▁▁▁▁▃▃██████▆▅▃▃▂▂▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂ ▃
+  5 ms           Histogram: frequency by time        9.66 ms <
 
  Memory estimate: 57.62 MiB, allocs estimate: 1011.
@benchmark jacobian($f_sparse_vector, $sparse_first_order_backend, $(randn(n)), $jac_extras_sparse) evals=1
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
- Range (min … max):  58.318 μs …   8.492 ms  ┊ GC (min … max):  0.00% … 18.43%
- Time  (median):     64.140 μs               ┊ GC (median):     0.00%
- Time  (mean ± σ):   73.458 μs ± 157.545 μs  ┊ GC (mean ± σ):  10.54% ±  6.50%
+ Range (min … max):  60.193 μs …   8.810 ms  ┊ GC (min … max):  0.00% … 21.60%
+ Time  (median):     66.510 μs               ┊ GC (median):     0.00%
+ Time  (mean ± σ):   77.138 μs ± 173.054 μs  ┊ GC (mean ± σ):  11.82% ±  6.54%
 
-           ▃▅▆▇█▇▆▃▂                                            
-  ▁▁▁▁▁▂▃▅██████████▇▆▄▄▃▃▂▂▂▂▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▃
-  58.3 μs         Histogram: frequency by time         83.3 μs <
+            ▁▃▅▆▇█▆▅▂                                           
+  ▁▁▁▁▂▂▃▄▆▇██████████▆▅▄▃▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▃
+  60.2 μs         Histogram: frequency by time         84.9 μs <
 
  Memory estimate: 399.66 KiB, allocs estimate: 26.
+