From efd7a221bf12c9a816f62542c5ded5525323acfc Mon Sep 17 00:00:00 2001 From: "Documenter.jl" Date: Wed, 17 Jul 2024 07:23:56 +0000 Subject: [PATCH] build based on a86ebb8 --- .../dev/.documenter-siteinfo.json | 2 +- DifferentiationInterface/dev/api/index.html | 52 +++++++++---------- .../dev/backends/index.html | 2 +- .../dev/dev_guide/index.html | 2 +- DifferentiationInterface/dev/index.html | 2 +- .../dev/operators/index.html | 2 +- .../dev/overloads/index.html | 46 ++++++++-------- .../dev/tutorial1/index.html | 52 +++++++++---------- .../dev/tutorial2/index.html | 28 +++++----- 9 files changed, 94 insertions(+), 94 deletions(-) diff --git a/DifferentiationInterface/dev/.documenter-siteinfo.json b/DifferentiationInterface/dev/.documenter-siteinfo.json index e8ab5bcfc..1f1bb8627 100644 --- a/DifferentiationInterface/dev/.documenter-siteinfo.json +++ b/DifferentiationInterface/dev/.documenter-siteinfo.json @@ -1 +1 @@ -{"documenter":{"julia_version":"1.10.4","generation_timestamp":"2024-07-17T07:13:10","documenter_version":"1.5.0"}} \ No newline at end of file +{"documenter":{"julia_version":"1.10.4","generation_timestamp":"2024-07-17T07:23:49","documenter_version":"1.5.0"}} \ No newline at end of file diff --git a/DifferentiationInterface/dev/api/index.html b/DifferentiationInterface/dev/api/index.html index a48c053f4..d1ee190b9 100644 --- a/DifferentiationInterface/dev/api/index.html +++ b/DifferentiationInterface/dev/api/index.html @@ -1,27 +1,27 @@ -API · DifferentiationInterface.jl
GitHub

API

DifferentiationInterface — Module
DifferentiationInterface

An interface to various automatic differentiation backends in Julia.

Exports

source

First order

Pushforward

DifferentiationInterface.prepare_pushforward — Function
prepare_pushforward(f,     backend, x, dx) -> extras
-prepare_pushforward(f!, y, backend, x, dx) -> extras

Create an extras object that can be given to pushforward and its variants.

Warning

If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.

source
DifferentiationInterface.prepare_pushforward_same_point — Function
prepare_pushforward_same_point(f,     backend, x, dx) -> extras_same
-prepare_pushforward_same_point(f!, y, backend, x, dx) -> extras_same

Create an extras_same object that can be given to pushforward and its variants if they are applied at the same point x.

Warning

If the function or the point changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.

source
DifferentiationInterface.pushforward! — Function
pushforward!(f,     dy, backend, x, dx, [extras]) -> dy
-pushforward!(f!, y, dy, backend, x, dx, [extras]) -> dy

Compute the pushforward of the function f at point x with seed dx, overwriting dy.

To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.

Tip

Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named jvp!.

source
DifferentiationInterface.value_and_pushforward — Function
value_and_pushforward(f,     backend, x, dx, [extras]) -> (y, dy)
-value_and_pushforward(f!, y, backend, x, dx, [extras]) -> (y, dy)

Compute the value and the pushforward of the function f at point x with seed dx.

To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.

Tip

Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named value_and_jvp.

Info

Required primitive for forward mode backends.

source
DifferentiationInterface.value_and_pushforward! — Function
value_and_pushforward!(f,     dy, backend, x, dx, [extras]) -> (y, dy)
-value_and_pushforward!(f!, y, dy, backend, x, dx, [extras]) -> (y, dy)

Compute the value and the pushforward of the function f at point x with seed dx, overwriting dy.

To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.

Tip

Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named value_and_jvp!.

source

Pullback

DifferentiationInterface.prepare_pullback — Function
prepare_pullback(f,     backend, x, dy) -> extras
-prepare_pullback(f!, y, backend, x, dy) -> extras

Create an extras object that can be given to pullback and its variants.

Warning

If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.

source
DifferentiationInterface.prepare_pullback_same_point — Function
prepare_pullback_same_point(f,     backend, x, dy) -> extras_same
-prepare_pullback_same_point(f!, y, backend, x, dy) -> extras_same

Create an extras_same object that can be given to pullback and its variants if they are applied at the same point x.

Warning

If the function or the point changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.

source
DifferentiationInterface.pullback — Function
pullback(f,     backend, x, dy, [extras]) -> dx
-pullback(f!, y, backend, x, dy, [extras]) -> dx

Compute the pullback of the function f at point x with seed dy.

To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.

Tip

Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named vjp.

source
DifferentiationInterface.pullback! — Function
pullback!(f,     dx, backend, x, dy, [extras]) -> dx
-pullback!(f!, y, dx, backend, x, dy, [extras]) -> dx

Compute the pullback of the function f at point x with seed dy, overwriting dx.

To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.

Tip

Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named vjp!.

source
DifferentiationInterface.value_and_pullback — Function
value_and_pullback(f,     backend, x, dy, [extras]) -> (y, dx)
-value_and_pullback(f!, y, backend, x, dy, [extras]) -> (y, dx)

Compute the value and the pullback of the function f at point x with seed dy.

To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.

Tip

Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named value_and_vjp.

Info

Required primitive for reverse mode backends.

source
DifferentiationInterface.value_and_pullback! — Function
value_and_pullback!(f,     dx, backend, x, dy, [extras]) -> (y, dx)
-value_and_pullback!(f!, y, dx, backend, x, dy, [extras]) -> (y, dx)

Compute the value and the pullback of the function f at point x with seed dy, overwriting dx.

To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.

Tip

Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named value_and_vjp!.

source

Derivative

DifferentiationInterface.prepare_derivative — Function
prepare_derivative(f,     backend, x) -> extras
-prepare_derivative(f!, y, backend, x) -> extras

Create an extras object that can be given to derivative and its variants.

Warning

If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.

source
DifferentiationInterface.derivative! — Function
derivative!(f,     der, backend, x, [extras]) -> der
-derivative!(f!, y, der, backend, x, [extras]) -> der

Compute the derivative of the function f at point x, overwriting der.

To improve performance via operator preparation, refer to prepare_derivative.

source
DifferentiationInterface.value_and_derivative! — Function
value_and_derivative!(f,     der, backend, x, [extras]) -> (y, der)
-value_and_derivative!(f!, y, der, backend, x, [extras]) -> (y, der)

Compute the value and the derivative of the function f at point x, overwriting der.

To improve performance via operator preparation, refer to prepare_derivative.

source

Gradient

DifferentiationInterface.prepare_gradient — Function
prepare_gradient(f, backend, x) -> extras

Create an extras object that can be given to gradient and its variants.

Warning

If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.

source

Jacobian

DifferentiationInterface.prepare_jacobian — Function
prepare_jacobian(f,     backend, x) -> extras
-prepare_jacobian(f!, y, backend, x) -> extras

Create an extras object that can be given to jacobian and its variants.

Warning

If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.

source
DifferentiationInterface.jacobian! — Function
jacobian!(f,     jac, backend, x, [extras]) -> jac
-jacobian!(f!, y, jac, backend, x, [extras]) -> jac

Compute the Jacobian matrix of the function f at point x, overwriting jac.

To improve performance via operator preparation, refer to prepare_jacobian.

source
DifferentiationInterface.value_and_jacobian! — Function
value_and_jacobian!(f,     jac, backend, x, [extras]) -> (y, jac)
-value_and_jacobian!(f!, y, jac, backend, x, [extras]) -> (y, jac)

Compute the value and the Jacobian matrix of the function f at point x, overwriting jac.

To improve performance via operator preparation, refer to prepare_jacobian.

source

Second order

DifferentiationInterface.SecondOrder — Type
SecondOrder

Combination of two backends for second-order differentiation.

Danger

SecondOrder backends do not support first-order operators.

Constructor

SecondOrder(outer_backend, inner_backend)

Fields

  • outer::ADTypes.AbstractADType: backend for the outer differentiation

  • inner::ADTypes.AbstractADType: backend for the inner differentiation

source

Second derivative

Hessian-vector product

DifferentiationInterface.prepare_hvp — Function
prepare_hvp(f, backend, x, dx) -> extras

Create an extras object that can be given to hvp and its variants.

Warning

If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.

source
DifferentiationInterface.prepare_hvp_same_point — Function
prepare_hvp_same_point(f, backend, x, dx) -> extras_same

Create an extras_same object that can be given to hvp and its variants if they are applied at the same point x.

Warning

If the function or the point changes in any way, the result of preparation will be invalidated, and you will need to run it again.

source

Hessian

DifferentiationInterface.prepare_hessian — Function
prepare_hessian(f, backend, x) -> extras

Create an extras object that can be given to hessian and its variants.

Warning

If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.

source

Utilities

Backend queries

DifferentiationInterface.check_hessian — Function
check_hessian(backend)

Check whether backend supports second order differentiation by trying to compute a hessian.

Warning

Might take a while due to compilation time.

source

Backend switch

DifferentiationInterface.DifferentiateWith — Type
DifferentiateWith

Callable function wrapper that enforces differentiation with a specified (inner) backend.

This works by defining new rules overriding the behavior of the outer backend that would normally be used.

Warning

This is an experimental functionality, whose API cannot yet be considered stable. At the moment, it only supports one-argument functions, and rules are only defined for ChainRules.jl-compatible outer backends.

Fields

  • f: the function in question
  • backend::AbstractADType: the inner backend to use for differentiation

Constructor

DifferentiateWith(f, backend)

Example

using DifferentiationInterface
+API · DifferentiationInterface.jl
GitHub
API
DifferentiationInterface — Module
DifferentiationInterface
An interface to various automatic differentiation backends in Julia.
Exports
source
First order
Pushforward
DifferentiationInterface.prepare_pushforward — Function
prepare_pushforward(f,     backend, x, dx) -> extras
+prepare_pushforward(f!, y, backend, x, dx) -> extras
Create an extras object that can be given to pushforward and its variants.
Warning
If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.
source
DifferentiationInterface.prepare_pushforward_same_point — Function
prepare_pushforward_same_point(f,     backend, x, dx) -> extras_same
+prepare_pushforward_same_point(f!, y, backend, x, dx) -> extras_same
Create an extras_same object that can be given to pushforward and its variants if they are applied at the same point x.
Warning
If the function or the point changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.
source
DifferentiationInterface.pushforward! — Function
pushforward!(f,     dy, backend, x, dx, [extras]) -> dy
+pushforward!(f!, y, dy, backend, x, dx, [extras]) -> dy
Compute the pushforward of the function f at point x with seed dx, overwriting dy.
To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.
Tip
Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named jvp!.
source
DifferentiationInterface.value_and_pushforward — Function
value_and_pushforward(f,     backend, x, dx, [extras]) -> (y, dy)
+value_and_pushforward(f!, y, backend, x, dx, [extras]) -> (y, dy)
Compute the value and the pushforward of the function f at point x with seed dx.
To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.
Tip
Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named value_and_jvp.
Info
Required primitive for forward mode backends.
source
DifferentiationInterface.value_and_pushforward! — Function
value_and_pushforward!(f,     dy, backend, x, dx, [extras]) -> (y, dy)
+value_and_pushforward!(f!, y, dy, backend, x, dx, [extras]) -> (y, dy)
Compute the value and the pushforward of the function f at point x with seed dx, overwriting dy.
To improve performance via operator preparation, refer to prepare_pushforward and prepare_pushforward_same_point.
Tip
Pushforwards are also commonly called Jacobian-vector products or JVPs. This function could have been named value_and_jvp!.
source
Pullback
DifferentiationInterface.prepare_pullback — Function
prepare_pullback(f,     backend, x, dy) -> extras
+prepare_pullback(f!, y, backend, x, dy) -> extras
Create an extras object that can be given to pullback and its variants.
Warning
If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.
source
DifferentiationInterface.prepare_pullback_same_point — Function
prepare_pullback_same_point(f,     backend, x, dy) -> extras_same
+prepare_pullback_same_point(f!, y, backend, x, dy) -> extras_same
Create an extras_same object that can be given to pullback and its variants if they are applied at the same point x.
Warning
If the function or the point changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.
source
DifferentiationInterface.pullback — Function
pullback(f,     backend, x, dy, [extras]) -> dx
+pullback(f!, y, backend, x, dy, [extras]) -> dx
Compute the pullback of the function f at point x with seed dy.
To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.
Tip
Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named vjp.
source
DifferentiationInterface.pullback! — Function
pullback!(f,     dx, backend, x, dy, [extras]) -> dx
+pullback!(f!, y, dx, backend, x, dy, [extras]) -> dx
Compute the pullback of the function f at point x with seed dy, overwriting dx.
To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.
Tip
Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named vjp!.
source
DifferentiationInterface.value_and_pullback — Function
value_and_pullback(f,     backend, x, dy, [extras]) -> (y, dx)
+value_and_pullback(f!, y, backend, x, dy, [extras]) -> (y, dx)
Compute the value and the pullback of the function f at point x with seed dy.
To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.
Tip
Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named value_and_vjp.
Info
Required primitive for reverse mode backends.
source
DifferentiationInterface.value_and_pullback! — Function
value_and_pullback!(f,     dx, backend, x, dy, [extras]) -> (y, dx)
+value_and_pullback!(f!, y, dx, backend, x, dy, [extras]) -> (y, dx)
Compute the value and the pullback of the function f at point x with seed dy, overwriting dx.
To improve performance via operator preparation, refer to prepare_pullback and prepare_pullback_same_point.
Tip
Pullbacks are also commonly called vector-Jacobian products or VJPs. This function could have been named value_and_vjp!.
source
Derivative
DifferentiationInterface.prepare_derivative — Function
prepare_derivative(f,     backend, x) -> extras
+prepare_derivative(f!, y, backend, x) -> extras
Create an extras object that can be given to derivative and its variants.
Warning
If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.
source
DifferentiationInterface.derivative! — Function
derivative!(f,     der, backend, x, [extras]) -> der
+derivative!(f!, y, der, backend, x, [extras]) -> der
Compute the derivative of the function f at point x, overwriting der.
To improve performance via operator preparation, refer to prepare_derivative.
source
DifferentiationInterface.value_and_derivative! — Function
value_and_derivative!(f,     der, backend, x, [extras]) -> (y, der)
+value_and_derivative!(f!, y, der, backend, x, [extras]) -> (y, der)
Compute the value and the derivative of the function f at point x, overwriting der.
To improve performance via operator preparation, refer to prepare_derivative.
source
Gradient
DifferentiationInterface.prepare_gradient — Function
prepare_gradient(f, backend, x) -> extras
Create an extras object that can be given to gradient and its variants.
Warning
If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.
source
Jacobian
DifferentiationInterface.prepare_jacobian — Function
prepare_jacobian(f,     backend, x) -> extras
+prepare_jacobian(f!, y, backend, x) -> extras
Create an extras object that can be given to jacobian and its variants.
Warning
If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again. In the two-argument case, y is mutated by f! during preparation.
source
DifferentiationInterface.jacobian! — Function
jacobian!(f,     jac, backend, x, [extras]) -> jac
+jacobian!(f!, y, jac, backend, x, [extras]) -> jac
Compute the Jacobian matrix of the function f at point x, overwriting jac.
To improve performance via operator preparation, refer to prepare_jacobian.
source
DifferentiationInterface.value_and_jacobian! — Function
value_and_jacobian!(f,     jac, backend, x, [extras]) -> (y, jac)
+value_and_jacobian!(f!, y, jac, backend, x, [extras]) -> (y, jac)
Compute the value and the Jacobian matrix of the function f at point x, overwriting jac.
To improve performance via operator preparation, refer to prepare_jacobian.
source
Second order
DifferentiationInterface.SecondOrder — Type
SecondOrder
Combination of two backends for second-order differentiation.
Danger
SecondOrder backends do not support first-order operators.
Constructor
SecondOrder(outer_backend, inner_backend)
Fields
  • outer::ADTypes.AbstractADType: backend for the outer differentiation
  • inner::ADTypes.AbstractADType: backend for the inner differentiation
source
Second derivative
Hessian-vector product
DifferentiationInterface.prepare_hvp — Function
prepare_hvp(f, backend, x, dx) -> extras
Create an extras object that can be given to hvp and its variants.
Warning
If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.
source
DifferentiationInterface.prepare_hvp_same_point — Function
prepare_hvp_same_point(f, backend, x, dx) -> extras_same
Create an extras_same object that can be given to hvp and its variants if they are applied at the same point x.
Warning
If the function or the point changes in any way, the result of preparation will be invalidated, and you will need to run it again.
source
Hessian
DifferentiationInterface.prepare_hessian — Function
prepare_hessian(f, backend, x) -> extras
Create an extras object that can be given to hessian and its variants.
Warning
If the function changes in any way, the result of preparation will be invalidated, and you will need to run it again.
source
Utilities
Backend queries
DifferentiationInterface.check_hessian — Function
check_hessian(backend)
Check whether backend supports second order differentiation by trying to compute a hessian.
Warning
Might take a while due to compilation time.
source
Backend switch
DifferentiationInterface.DifferentiateWith — Type
DifferentiateWith
Callable function wrapper that enforces differentiation with a specified (inner) backend.
This works by defining new rules overriding the behavior of the outer backend that would normally be used.
Warning
This is an experimental functionality, whose API cannot yet be considered stable. At the moment, it only supports one-argument functions, and rules are only defined for ChainRules.jl-compatible outer backends.
Fields
  • f: the function in question
  • backend::AbstractADType: the inner backend to use for differentiation
Constructor
DifferentiateWith(f, backend)
Example
using DifferentiationInterface
 import ForwardDiff, Zygote
 
 function f(x)
@@ -37,7 +37,7 @@
 # output
 
 1-element Vector{Float64}:
- 1.0
source
Sparsity detection
DifferentiationInterface.DenseSparsityDetector — Type
DenseSparsityDetector
Sparsity pattern detector satisfying the detection API of ADTypes.jl.
The nonzeros in a Jacobian or Hessian are detected by computing the relevant matrix with dense AD, and thresholding the entries with a given tolerance (which can be numerically inaccurate).
Warning
This detector can be very slow, and should only be used if its output can be exploited multiple times to compute many sparse matrices. 
Danger
In general, the sparsity pattern you obtain can depend on the provided input x. If you want to reuse the pattern, make sure that it is input-agnostic.
Fields
  • backend::AbstractADType is the dense AD backend used under the hood
  • atol::Float64 is the minimum magnitude of a matrix entry to be considered nonzero
Constructor
DenseSparsityDetector(backend; atol, method=:iterative)
The keyword argument method::Symbol can be either:
  • :iterative: compute the matrix in a sequence of matrix-vector products (memory-efficient)
  • :direct: compute the matrix all at once (memory-hungry but sometimes faster).
Note that the constructor is type-unstable because method ends up being a type parameter of the DenseSparsityDetector object (this is not part of the API and might change).
Examples
using ADTypes, DifferentiationInterface, SparseArrays
+ 1.0
source
Sparsity detection
DifferentiationInterface.DenseSparsityDetector — Type
DenseSparsityDetector
Sparsity pattern detector satisfying the detection API of ADTypes.jl.
The nonzeros in a Jacobian or Hessian are detected by computing the relevant matrix with dense AD, and thresholding the entries with a given tolerance (which can be numerically inaccurate).
Warning
This detector can be very slow, and should only be used if its output can be exploited multiple times to compute many sparse matrices. 
Danger
In general, the sparsity pattern you obtain can depend on the provided input x. If you want to reuse the pattern, make sure that it is input-agnostic.
Fields
  • backend::AbstractADType is the dense AD backend used under the hood
  • atol::Float64 is the minimum magnitude of a matrix entry to be considered nonzero
Constructor
DenseSparsityDetector(backend; atol, method=:iterative)
The keyword argument method::Symbol can be either:
  • :iterative: compute the matrix in a sequence of matrix-vector products (memory-efficient)
  • :direct: compute the matrix all at once (memory-hungry but sometimes faster).
Note that the constructor is type-unstable because method ends up being a type parameter of the DenseSparsityDetector object (this is not part of the API and might change).
Examples
using ADTypes, DifferentiationInterface, SparseArrays
 import ForwardDiff
 
 detector = DenseSparsityDetector(AutoForwardDiff(); atol=1e-5, method=:direct)
@@ -60,9 +60,9 @@
 # output
 
 1×2 SparseMatrixCSC{Bool, Int64} with 1 stored entry:
- 1  ⋅
source
Internals
The following is not part of the public API.
DifferentiationInterface.Batch — Type
Batch{B,T}
Efficient storage for B elements of type T (NTuple wrapper).
A Batch can be used as seed to trigger batched-mode pushforward, pullback and hvp.
Fields
  • elements::NTuple{B,T}
source
ADTypes.mode — Method
mode(backend::SecondOrder)
Return the outer mode of the second-order backend.
source
DifferentiationInterface.basis — Method
basis(backend, a::AbstractArray, i::CartesianIndex)
Construct the i-th stardard basis array in the vector space of a with element type eltype(a).
Note
If an AD backend benefits from a more specialized basis array implementation, this function can be extended on the backend type.
source
DifferentiationInterface.nested — Method
nested(backend)
Return a possibly modified backend that can work while nested inside another differentiation procedure.
At the moment, this is only useful for Enzyme, which needs autodiff_deferred to be compatible with higher-order differentiation.
source
DifferentiationInterface.pick_batchsize — Method
pick_batchsize(backend::AbstractADType, dimension::Integer)
Pick a reasonable batch size for batched derivative evaluation with a given total dimension.
Returns 1 for backends which have not overloaded it.
source

+
diff --git a/DifferentiationInterface/dev/backends/index.html b/DifferentiationInterface/dev/backends/index.html index e6933d1e5..1f79832ba 100644 --- a/DifferentiationInterface/dev/backends/index.html +++ b/DifferentiationInterface/dev/backends/index.html @@ -5,4 +5,4 @@ startOnLoad: true, theme: "neutral" }); - + diff --git a/DifferentiationInterface/dev/dev_guide/index.html b/DifferentiationInterface/dev/dev_guide/index.html index 3514b187d..f8306e415 100644 --- a/DifferentiationInterface/dev/dev_guide/index.html +++ b/DifferentiationInterface/dev/dev_guide/index.html @@ -4,4 +4,4 @@ startOnLoad: true, theme: "neutral" }); - + diff --git a/DifferentiationInterface/dev/index.html b/DifferentiationInterface/dev/index.html index 94db5cac1..2b8e1e0ba 100644 --- a/DifferentiationInterface/dev/index.html +++ b/DifferentiationInterface/dev/index.html @@ -20,4 +20,4 @@ startOnLoad: true, theme: "neutral" }); - + diff --git a/DifferentiationInterface/dev/operators/index.html b/DifferentiationInterface/dev/operators/index.html index 2ffb44740..a1724678b 100644 --- a/DifferentiationInterface/dev/operators/index.html +++ b/DifferentiationInterface/dev/operators/index.html @@ -5,4 +5,4 @@ startOnLoad: true, theme: "neutral" }); - + diff --git a/DifferentiationInterface/dev/overloads/index.html b/DifferentiationInterface/dev/overloads/index.html index 6fef42b0c..f0e8d1e0e 100644 --- a/DifferentiationInterface/dev/overloads/index.html +++ b/DifferentiationInterface/dev/overloads/index.html @@ -1,54 +1,54 @@ Overloads · DifferentiationInterface.jl
GitHub

Overloads

DifferentiationInterface.jl provides a handful of operators like gradient or jacobian, each with several variants:

  • out-of-place or in-place behavior
  • with or without primal output value
  • support for one-argument functions y = f(x) or two-argument functions f!(y, x)

While it is possible to define every operator using just pushforward and pullback, some backends have more efficient implementations of high-level operators. When they are available, we always call these backend-specific overloads.

The following tables summarize all implemented overloads for each backend. The cells can have three values:

  • ❌: the operator is not overloaded because the backend does not support it
  • ✅: the operator is overloaded
  • NA: the operator does not exist
Tip

Check marks (✅) are clickable and link to the source code.

Diffractor (forward/reverse)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient❌❌❌❌
jacobian❌❌❌❌
hessian❌❌NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅❌✅❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient❌❌❌❌
jacobian❌❌❌❌
hessian❌❌NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅❌✅❌

Two-argument functions f!(y, x)

Backend doesn't support mutating functions.

Enzyme (forward)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian❌❌NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅✅✅✅
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian❌❌NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅✅✅✅

Two-argument functions f!(y, x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
jacobian❌❌❌❌
pullback❌❌❌❌
pushforward❌❌✅❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
jacobian❌❌❌❌
pullback❌❌❌❌
pushforward❌❌✅❌

Enzyme (reverse)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian❌❌❌❌
hessian❌❌NANA
hvp❌❌NANA
pullback✅✅✅✅
pushforward❌❌❌❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian❌❌❌❌
hessian❌❌NANA
hvp❌❌NANA
pullback✅✅✅✅
pushforward❌❌❌❌

Two-argument functions f!(y, x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
jacobian❌❌❌❌
pullback❌❌✅❌
pushforward❌❌❌❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
jacobian❌❌❌❌
pullback❌❌✅❌
pushforward❌❌❌❌

FastDifferentiation (symbolic)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp✅✅NANA
pullback✅✅✅✅
pushforward✅✅✅✅
+
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp✅✅NANA
pullback✅✅✅✅
pushforward✅✅✅✅

Two-argument functions f!(y, x)

-
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback✅✅✅✅
pushforward✅✅✅✅
+
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback✅✅✅✅
pushforward✅✅✅✅

FiniteDiff (forward)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅❌✅❌
+
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅❌✅❌

Two-argument functions f!(y, x)

-
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback❌❌❌❌
pushforward❌❌✅❌
+
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback❌❌❌❌
pushforward❌❌✅❌

FiniteDifferences (forward)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian❌❌NANA
hvp❌❌NANA
pullback✅❌✅❌
pushforward✅❌✅❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian❌❌NANA
hvp❌❌NANA
pullback✅❌✅❌
pushforward✅❌✅❌

Two-argument functions f!(y, x)

Backend doesn't support mutating functions.

ForwardDiff (forward)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅✅✅✅
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅✅✅✅

Two-argument functions f!(y, x)

-
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback❌❌❌❌
pushforward✅✅✅✅
+
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback❌❌❌❌
pushforward✅✅✅✅

PolyesterForwardDiff (forward)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅✅✅✅
+
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅✅✅✅

Two-argument functions f!(y, x)

-
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback❌❌❌❌
pushforward✅✅✅✅
+
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback❌❌❌❌
pushforward✅✅✅✅

ReverseDiff (reverse)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌✅✅
pushforward❌❌❌❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌✅✅
pushforward❌❌❌❌

Two-argument functions f!(y, x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
jacobian✅✅✅✅
pullback✅✅✅✅
pushforward❌❌❌❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
jacobian✅✅✅✅
pullback✅✅✅✅
pushforward❌❌❌❌

Symbolics (symbolic)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅✅✅✅
+
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp❌❌NANA
pullback❌❌❌❌
pushforward✅✅✅✅

Two-argument functions f!(y, x)

-
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback❌❌❌❌
pushforward✅✅✅✅
+
Operatoropop!value_and_opvalue_and_op!
derivative✅✅✅✅
jacobian✅✅✅✅
pullback❌❌❌❌
pushforward✅✅✅✅

Tapir (reverse)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient❌❌❌❌
jacobian❌❌❌❌
hessian❌❌NANA
hvp❌❌NANA
pullback✅✅✅✅
pushforward❌❌❌❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient❌❌❌❌
jacobian❌❌❌❌
hessian❌❌NANA
hvp❌❌NANA
pullback✅✅✅✅
pushforward❌❌❌❌

Two-argument functions f!(y, x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
jacobian❌❌❌❌
pullback❌❌✅❌
pushforward❌❌❌❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
jacobian❌❌❌❌
pullback❌❌✅❌
pushforward❌❌❌❌

Tracker (reverse)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian❌❌❌❌
hessian❌❌NANA
hvp❌❌NANA
pullback✅❌✅❌
pushforward❌❌❌❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian❌❌❌❌
hessian❌❌NANA
hvp❌❌NANA
pullback✅❌✅❌
pushforward❌❌❌❌

Two-argument functions f!(y, x)

Backend doesn't support mutating functions.

Zygote (reverse)

One-argument functions y = f(x)

-
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp✅✅NANA
pullback✅❌✅❌
pushforward❌❌❌❌
+
Operatoropop!value_and_opvalue_and_op!
derivative❌❌❌❌
gradient✅✅✅✅
jacobian✅✅✅✅
hessian✅✅NANA
hvp✅✅NANA
pullback✅❌✅❌
pushforward❌❌❌❌

Two-argument functions f!(y, x)

Backend doesn't support mutating functions.

+ diff --git a/DifferentiationInterface/dev/tutorial1/index.html b/DifferentiationInterface/dev/tutorial1/index.html index e3d21819c..edb035664 100644 --- a/DifferentiationInterface/dev/tutorial1/index.html +++ b/DifferentiationInterface/dev/tutorial1/index.html @@ -21,14 +21,14 @@ 8.0 10.0

Was that fast? BenchmarkTools.jl helps you answer that question.

using BenchmarkTools
 
-@benchmark gradient($f, $backend, $x)
BenchmarkTools.Trial: 10000 samples with 182 evaluations.
- Range (min … max):  571.571 ns …  1.750 ms  ┊ GC (min … max):  0.00% … 99.95%
- Time  (median):     593.585 ns              ┊ GC (median):     0.00%
- Time  (mean ± σ):   887.801 ns ± 17.508 μs  ┊ GC (mean ± σ):  22.11% ±  3.90%
+@benchmark gradient($f, $backend, $x)
BenchmarkTools.Trial: 10000 samples with 185 evaluations.
+ Range (min … max):  566.351 ns …  1.792 ms  ┊ GC (min … max):  0.00% … 99.94%
+ Time  (median):     587.746 ns              ┊ GC (median):     0.00%
+ Time  (mean ± σ):   866.596 ns ± 17.924 μs  ┊ GC (mean ± σ):  23.14% ±  3.94%
 
-  ▄▇█▇▄▂▁   ▁                               ▂▄▄▃▃▄▅▅▄▂         ▂
-  ███████▆▇████▇▆▆▆▆▆▆▄▆▃▁▅▁▄▃▁▁▁▄▄▃▄▅▄▃▆▅▆▇███████████▇▇▆▇█▇▆ █
-  572 ns        Histogram: log(frequency) by time         1 μs <
+  ▅▆█▇▅▂▁    ▁                                ▁▃▃▃▂▃▄▄▄▃▂      ▂
+  ███████▇▇▇████▆▆▅▆▅▆▆▅▆▆▄▄▃▄▄▄▁▄▃▃▃▃▁▃▁▁▃▁▄▅███████████▇▆▆▇▆ █
+  566 ns        Histogram: log(frequency) by time       987 ns <
 
  Memory estimate: 848 bytes, allocs estimate: 4.

Not bad, but you can do better.

Overwriting a gradient

Since you know how much space your gradient will occupy (the same as your input x), you can pre-allocate that memory and offer it to AD. Some backends get a speed boost from this trick.

grad = similar(x)
 gradient!(f, grad, backend, x)
@@ -38,15 +38,15 @@
   6.0
   8.0
  10.0

The bang indicates that one of the arguments of gradient! might be mutated. More precisely, our convention is that every positional argument between the function and the backend is mutated (and the extras too, see below).

@benchmark gradient!($f, _grad, $backend, $x) evals=1 setup=(_grad=similar($x))
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
- Range (min … max):  570.000 ns …  10.049 μs  ┊ GC (min … max): 0.00% … 0.00%
- Time  (median):     611.000 ns               ┊ GC (median):    0.00%
- Time  (mean ± σ):   645.001 ns ± 137.264 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%
+ Range (min … max):  571.000 ns …  10.760 μs  ┊ GC (min … max): 0.00% … 0.00%
+ Time  (median):     621.000 ns               ┊ GC (median):    0.00%
+ Time  (mean ± σ):   669.192 ns ± 197.433 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%
 
-     ██▂                                                         
-  ▃█▃███▄▇▅▁▄▄▁▄▄▁▄▃▃▂▃▃▂▃▃▁▃▃▁▃▃▁▃▃▂▂▂▂▁▂▂▁▂▂▁▂▂▂▂▂▂▂▂▂▁▂▂▁▂▂▂ ▃
-  570 ns           Histogram: frequency by time          991 ns <
+     ▂ ██▁                                                       
+  ▂▂▄█▂███▄▃▃▃▃▃▂▃▄▄▁▃▃▃▃▁▃▃▃▂▂▃▃▂▂▂▂▂▂▁▂▂▂▂▁▂▂▂▂▂▂▂▂▂▂▂▂▂▁▂▂▂▂ ▃
+  571 ns           Histogram: frequency by time         1.05 μs <
 
- Memory estimate: 752 bytes, allocs estimate: 3.

For some reason the in-place version is not much better than your first attempt. However, it makes fewer allocations, thanks to the gradient vector you provided. Don't worry, you can get even more performance.

Preparing for multiple gradients

Internally, ForwardDiff.jl creates some data structures to keep track of things. These objects can be reused between gradient computations, even on different input values. We abstract away the preparation step behind a backend-agnostic syntax:

extras = prepare_gradient(f, backend, randn(eltype(x), size(x)))
DifferentiationInterfaceForwardDiffExt.ForwardDiffGradientExtras{ForwardDiff.GradientConfig{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5, Vector{ForwardDiff.Dual{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5}}}}(ForwardDiff.GradientConfig{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5, Vector{ForwardDiff.Dual{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5}}}((Partials(1.0, 0.0, 0.0, 0.0, 0.0), Partials(0.0, 1.0, 0.0, 0.0, 0.0), Partials(0.0, 0.0, 1.0, 0.0, 0.0), Partials(0.0, 0.0, 0.0, 1.0, 0.0), Partials(0.0, 0.0, 0.0, 0.0, 1.0)), ForwardDiff.Dual{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5}[Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(0.0,0.0,0.0,0.0,0.0,0.0), Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(0.0,0.0,0.0,0.0,0.0,0.0), Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(0.0,0.0,0.0,0.0,0.0,0.0), Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(0.0,0.0,0.0,0.0,0.0,0.0), Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(0.0,0.0,0.0,0.0,0.0,0.0)]))

You don't need to know what this object is, you just need to pass it to the gradient operator. Note that preparation does not depend on the actual components of the vector x, just on its type and size. You can thus reuse the extras for different values of the input.

grad = similar(x)
+ Memory estimate: 752 bytes, allocs estimate: 3.

For some reason the in-place version is not much better than your first attempt. However, it makes fewer allocations, thanks to the gradient vector you provided. Don't worry, you can get even more performance.

Preparing for multiple gradients

Internally, ForwardDiff.jl creates some data structures to keep track of things. These objects can be reused between gradient computations, even on different input values. We abstract away the preparation step behind a backend-agnostic syntax:

extras = prepare_gradient(f, backend, randn(eltype(x), size(x)))
DifferentiationInterfaceForwardDiffExt.ForwardDiffGradientExtras{ForwardDiff.GradientConfig{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5, Vector{ForwardDiff.Dual{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5}}}}(ForwardDiff.GradientConfig{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5, Vector{ForwardDiff.Dual{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5}}}((Partials(1.0, 0.0, 0.0, 0.0, 0.0), Partials(0.0, 1.0, 0.0, 0.0, 0.0), Partials(0.0, 0.0, 1.0, 0.0, 0.0), Partials(0.0, 0.0, 0.0, 1.0, 0.0), Partials(0.0, 0.0, 0.0, 0.0, 1.0)), ForwardDiff.Dual{ForwardDiff.Tag{typeof(Main.f), Float64}, Float64, 5}[Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(1.0,1.0,0.0,0.0,0.0,0.0), Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(2.0,0.0,1.0,0.0,0.0,0.0), Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(3.0,0.0,0.0,1.0,0.0,0.0), Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(4.0,0.0,0.0,0.0,1.0,0.0), Dual{ForwardDiff.Tag{typeof(Main.f), Float64}}(5.0,0.0,0.0,0.0,0.0,1.0)]))

You don't need to know what this object is, you just need to pass it to the gradient operator. Note that preparation does not depend on the actual components of the vector x, just on its type and size. You can thus reuse the extras for different values of the input.

grad = similar(x)
 gradient!(f, grad, backend, x, extras)
 grad  # has been mutated
5-element Vector{Float64}:
   2.0
@@ -57,13 +57,13 @@
     _grad=similar($x);
     _extras=prepare_gradient($f, $backend, $x)
 )
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
- Range (min … max):  61.000 ns …  1.232 μs  ┊ GC (min … max): 0.00% … 0.00%
- Time  (median):     80.000 ns              ┊ GC (median):    0.00%
- Time  (mean ± σ):   84.947 ns ± 33.077 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%
+ Range (min … max):  60.000 ns …  2.805 μs  ┊ GC (min … max): 0.00% … 0.00%
+ Time  (median):     71.000 ns              ┊ GC (median):    0.00%
+ Time  (mean ± σ):   80.600 ns ± 39.047 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%
 
-     ▃█   ▇▄   ▅   ▃                  ▁   ▂▁   ▄   ▂▁   ▁     ▂
-  ▃▁▁██▁▁▁██▁▁▆█▁▁▁██▁▁▁█▁▁▁▆▃▁▁▁▄▃▁▁▁█▁▁▁██▁▁▁█▁▁▁██▁▁▁█▁▁▁▃ █
-  61 ns        Histogram: log(frequency) by time       190 ns <
+      █▅   ▇▅   ▄▂   ▁                        ▂    ▂    ▂     ▂
+  ▄▁▁▁██▁▁▁██▁▁▁██▁▁▁█▆▁▁▁▇▆▁▁▁▅▁▁▁▁▅▁▁▁▁█▁▁▁▁█▁▁▁▁█▁▁▁▁█▁▁▁▄ █
+  60 ns        Histogram: log(frequency) by time       180 ns <
 
  Memory estimate: 0 bytes, allocs estimate: 0.

Beware that the extras object is nearly always mutated by differentiation operators, even though it is given as the last positional argument.

Switching backends

The whole point of DifferentiationInterface.jl is that you can easily experiment with different AD solutions. Typically, for gradients, reverse mode AD might be a better fit, so let's try the state-of-the-art Enzyme.jl!

import Enzyme
 
@@ -76,17 +76,17 @@
     _grad=similar($x);
     _extras=prepare_gradient($f, $backend2, $x)
 )
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
- Range (min … max):  49.000 ns … 531.000 ns  ┊ GC (min … max): 0.00% … 0.00%
+ Range (min … max):  50.000 ns …  12.864 μs  ┊ GC (min … max): 0.00% … 0.00%
  Time  (median):     60.000 ns               ┊ GC (median):    0.00%
- Time  (mean ± σ):   63.006 ns ±  13.895 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%
+ Time  (mean ± σ):   65.538 ns ± 133.050 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%
 
-  ▁        █▅      ▅▃       ▂                                  ▁
-  █▇▁▁▁▁▁▁▁██▁▁▁▁▁███▁▁▁▁▁▁▁██▁▁▁▁▁▃█▆▁▁▁▁▁▁▁▆▁▁▁▁▁▁▁▆▃▁▁▁▁▁▁▆ █
-  49 ns         Histogram: log(frequency) by time       120 ns <
+          █▅      ▁▆       ▂                                   ▁
+  ▃▁▁▁▁▁▁▁██▁▁▁▁▁▁██▁▁▁▁▁▁▁██▁▁▁▁▁▁▃▇▅▁▁▁▁▁▁▆▄▁▁▁▁▁▁▁▆▄▁▁▁▁▁▁▅ █
+  50 ns         Histogram: log(frequency) by time       120 ns <
 
  Memory estimate: 0 bytes, allocs estimate: 0.

In short, DifferentiationInterface.jl allows for easy testing and comparison of AD backends. If you want to go further, check out the documentation of DifferentiationInterfaceTest.jl. This related package provides benchmarking utilities to compare backends and help you select the one that is best suited for your problem.

+ diff --git a/DifferentiationInterface/dev/tutorial2/index.html b/DifferentiationInterface/dev/tutorial2/index.html index cd06726e8..788747e79 100644 --- a/DifferentiationInterface/dev/tutorial2/index.html +++ b/DifferentiationInterface/dev/tutorial2/index.html @@ -57,27 +57,27 @@ -144.0 576.0 -432.0 ⋅ ⋅ ⋅ -1152.0 2560.0 -1440.0 160.0 -320.0 ⋅ ⋅ ⋅ ⋅ ⋅ -1440.0 1728.0

Sparse preparation

In the examples above, we didn't use preparation. Sparse preparation is more costly than dense preparation, but it is even more essential. Indeed, once preparation is done, sparse differentiation is much faster than dense differentiation, because it makes fewer calls to the underlying function. The speedup becomes very visible in large dimensions.

n = 1000
 jac_extras_dense = prepare_jacobian(f_sparse_vector, dense_first_order_backend, randn(n));
-jac_extras_sparse = prepare_jacobian(f_sparse_vector, sparse_first_order_backend, randn(n));
@benchmark jacobian($f_sparse_vector, $dense_first_order_backend, $(randn(n)), $jac_extras_dense) evals=1
BenchmarkTools.Trial: 726 samples with 1 evaluation.
- Range (min … max):  4.809 ms … 31.637 ms  ┊ GC (min … max):  0.00% … 51.69%
- Time  (median):     6.007 ms              ┊ GC (median):    17.41%
- Time  (mean ± σ):   5.926 ms ±  1.582 ms  ┊ GC (mean ± σ):  13.65% ±  8.55%
+jac_extras_sparse = prepare_jacobian(f_sparse_vector, sparse_first_order_backend, randn(n));
@benchmark jacobian($f_sparse_vector, $dense_first_order_backend, $(randn(n)), $jac_extras_dense) evals=1
BenchmarkTools.Trial: 653 samples with 1 evaluation.
+ Range (min … max):  4.878 ms … 37.491 ms  ┊ GC (min … max):  0.00% … 46.77%
+ Time  (median):     6.369 ms              ┊ GC (median):    19.06%
+ Time  (mean ± σ):   6.591 ms ±  2.085 ms  ┊ GC (mean ± σ):  15.38% ±  9.97%
 
-     ▄ ▃▂                           ▅▅▆█▄▃▂▃▃                 
-  ▃▃▇████▅▄▄▄▄▃▃▄▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▂▄▄▇█████████▆▅▅▅▄▄▄▅▃▅▄▃▁▃▂ ▄
-  4.81 ms        Histogram: frequency by time        6.71 ms <
+   █▂       ██▆▂                                              
+  ████▄▂▁▁▂▅████▆▅▅▅▅▄▄▄▃▃▃▃▁▁▁▂▁▁▃▂▃▃▁▃▃▃▃▁▁▁▂▂▁▁▁▁▁▂▁▁▁▁▁▂ ▃
+  4.88 ms        Histogram: frequency by time        12.4 ms <
 
  Memory estimate: 57.62 MiB, allocs estimate: 1011.
@benchmark jacobian($f_sparse_vector, $sparse_first_order_backend, $(randn(n)), $jac_extras_sparse) evals=1
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
- Range (min … max):  58.881 μs …   6.622 ms  ┊ GC (min … max): 0.00% … 98.05%
- Time  (median):     62.938 μs               ┊ GC (median):    0.00%
- Time  (mean ± σ):   70.303 μs ± 119.545 μs  ┊ GC (mean ± σ):  9.51% ±  6.55%
+ Range (min … max):  58.810 μs …   9.060 ms  ┊ GC (min … max):  0.00% … 17.37%
+ Time  (median):     63.434 μs               ┊ GC (median):     0.00%
+ Time  (mean ± σ):   72.728 μs ± 162.946 μs  ┊ GC (mean ± σ):  10.65% ±  6.53%
 
-       ▁▄▆▇█▇▆▅▄▄▂                                              
-  ▂▂▃▄▆████████████▆▅▄▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▁▂ ▄
-  58.9 μs         Histogram: frequency by time         80.7 μs <
+        ▁▄▆██▇▇▄▂▁                                              
+  ▁▁▁▂▄▆██████████▇▅▄▃▃▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▃
+  58.8 μs         Histogram: frequency by time           83 μs <
 
  Memory estimate: 399.66 KiB, allocs estimate: 26.
+