From accdd8a2f133d9a6a90a4475e124427392b475c6 Mon Sep 17 00:00:00 2001 From: "Documenter.jl" Date: Wed, 17 Jul 2024 07:20:09 +0000 Subject: [PATCH] build based on a86ebb8 --- .../dev/.documenter-siteinfo.json | 2 +- .../dev/api/index.html | 26 +++++++++---------- DifferentiationInterfaceTest/dev/index.html | 2 +- .../dev/tutorial/index.html | 14 +++++----- 4 files changed, 22 insertions(+), 22 deletions(-) diff --git a/DifferentiationInterfaceTest/dev/.documenter-siteinfo.json b/DifferentiationInterfaceTest/dev/.documenter-siteinfo.json index 44e3962fe..ef357069e 100644 --- a/DifferentiationInterfaceTest/dev/.documenter-siteinfo.json +++ b/DifferentiationInterfaceTest/dev/.documenter-siteinfo.json @@ -1 +1 @@ -{"documenter":{"julia_version":"1.10.4","generation_timestamp":"2024-07-17T07:09:47","documenter_version":"1.5.0"}} \ No newline at end of file +{"documenter":{"julia_version":"1.10.4","generation_timestamp":"2024-07-17T07:20:04","documenter_version":"1.5.0"}} \ No newline at end of file diff --git a/DifferentiationInterfaceTest/dev/api/index.html b/DifferentiationInterfaceTest/dev/api/index.html index 0a85a127a..df7bc41a2 100644 --- a/DifferentiationInterfaceTest/dev/api/index.html +++ b/DifferentiationInterfaceTest/dev/api/index.html @@ -1,5 +1,5 @@ -API reference · DifferentiationInterfaceTest.jl
GitHub

API reference

Entry points

DifferentiationInterfaceTest.test_differentiation — Function
test_differentiation(
+API reference · DifferentiationInterfaceTest.jl
GitHub
API reference
Entry points
DifferentiationInterfaceTest.test_differentiation — Function
test_differentiation(
     backends::Vector{<:ADTypes.AbstractADType};
     ...
 )
@@ -25,12 +25,12 @@
     atol,
     rtol
 )
-
Cross-test a list of backends on a list of scenarios, running a variety of different tests.
Default arguments
Keyword arguments
Testing:
  • correctness=true: whether to compare the differentiation results with the theoretical values specified in each scenario
  • type_stability=false: whether to check type stability with JET.jl (thanks to JET.@test_opt)
  • sparsity: whether to check sparsity of the jacobian / hessian
  • detailed=false: whether to print a detailed or condensed test log
Filtering:
  • input_type=Any, output_type=Any: restrict scenario inputs / outputs to subtypes of this
  • first_order=true, second_order=true: include first order / second order operators
  • onearg=true, twoarg=true: include one-argument / two-argument functions
  • inplace=true, outofplace=true: include in-place / out-of-place operators
Options:
  • logging=false: whether to log progress
  • isapprox=isapprox: function used to compare objects, with the standard signature isapprox(x, y; atol, rtol)
  • atol=0: absolute precision for correctness testing (when comparing to the reference outputs)
  • rtol=1e-3: relative precision for correctness testing (when comparing to the reference outputs)
source
test_differentiation(
+
Cross-test a list of backends on a list of scenarios, running a variety of different tests.
Default arguments
Keyword arguments
Testing:
  • correctness=true: whether to compare the differentiation results with the theoretical values specified in each scenario
  • type_stability=false: whether to check type stability with JET.jl (thanks to JET.@test_opt)
  • sparsity: whether to check sparsity of the jacobian / hessian
  • detailed=false: whether to print a detailed or condensed test log
Filtering:
  • input_type=Any, output_type=Any: restrict scenario inputs / outputs to subtypes of this
  • first_order=true, second_order=true: include first order / second order operators
  • onearg=true, twoarg=true: include one-argument / two-argument functions
  • inplace=true, outofplace=true: include in-place / out-of-place operators
Options:
  • logging=false: whether to log progress
  • isapprox=isapprox: function used to compare objects, with the standard signature isapprox(x, y; atol, rtol)
  • atol=0: absolute precision for correctness testing (when comparing to the reference outputs)
  • rtol=1e-3: relative precision for correctness testing (when comparing to the reference outputs)
source
test_differentiation(
     backend::ADTypes.AbstractADType,
     args...;
     kwargs...
 )
-
Shortcut for a single backend.
source
DifferentiationInterfaceTest.benchmark_differentiation — Function
benchmark_differentiation(
     backends::Vector{<:ADTypes.AbstractADType},
     scenarios::Vector{<:Scenario};
     input_type,
@@ -44,14 +44,14 @@
     excluded,
     logging
 ) -> DataFrames.DataFrame
-
Benchmark a list of backends for a list of operators on a list of scenarios.
The object returned is a DataFrames.DataFrame where each column corresponds to a field of DifferentiationBenchmarkDataRow.
The keyword arguments available here have the same meaning as those in test_differentiation.
source
DifferentiationInterfaceTest.DifferentiationBenchmarkDataRow — Type
DifferentiationBenchmarkDataRow
Ad-hoc storage type for differentiation benchmarking results.
If you have a vector rows::Vector{DifferentiationBenchmarkDataRow}, you can turn it into a DataFrame as follows:
using DataFrames
+
Benchmark a list of backends for a list of operators on a list of scenarios.
The object returned is a DataFrames.DataFrame where each column corresponds to a field of DifferentiationBenchmarkDataRow.
The keyword arguments available here have the same meaning as those in test_differentiation.
source
DifferentiationInterfaceTest.DifferentiationBenchmarkDataRow — Type
DifferentiationBenchmarkDataRow
Ad-hoc storage type for differentiation benchmarking results.
If you have a vector rows::Vector{DifferentiationBenchmarkDataRow}, you can turn it into a DataFrame as follows:
using DataFrames
 
-df = DataFrame(rows)
The resulting DataFrame will have one column for each of the following fields.
Fields
  • backend::ADTypes.AbstractADType: backend used for benchmarking
  • scenario::Scenario: scenario used for benchmarking
  • operator::Symbol: differentiation operator used for benchmarking, e.g. :gradient or :hessian
  • calls::Int64: number of calls to the differentiated function for one call to the operator
  • samples::Int64: number of benchmarking samples taken
  • evals::Int64: number of evaluations used for averaging in each sample
  • time::Float64: minimum runtime over all samples, in seconds
  • allocs::Float64: minimum number of allocations over all samples
  • bytes::Float64: minimum memory allocated over all samples, in bytes
  • gc_fraction::Float64: minimum fraction of time spent in garbage collection over all samples, between 0.0 and 1.0
  • compile_fraction::Float64: minimum fraction of time spent compiling over all samples, between 0.0 and 1.0
See the documentation of Chairmarks.jl for more details on the measurement fields.
source
Pre-made scenario lists
Scenario types
DifferentiationInterfaceTest.Scenario — Type
Scenario{op,args,pl}
Store a testing scenario composed of a function and its input + output for a given operator.
This generic type should never be used directly: use the specific constructor corresponding to the operator you want to test, or a predefined list of scenarios.
Constructors
Type parameters
  • op: one  of :pushforward, :pullback, :derivative, :gradient, :jacobian,:second_derivative, :hvp, :hessian
  • args: either 1 (for f(x) = y) or 2 (for f!(y, x) = nothing)
  • pl: either :inplace or :outofplace
Fields
  • f::Any: function f (if args==1) or f! (if args==2) to apply
  • x::Any: primal input
  • y::Any: primal output
  • seed::Any: seed for pushforward, pullback or HVP
  • res1::Any: first-order result of the operator
  • res2::Any: second-order result of the operator (when it makes sense)
Note that the res1 and res2 fields are given more meaningful names in the keyword arguments of each specialized constructor. For example:
  • the keyword grad of GradientScenario becomes res1
  • the keyword hess of HessianScenario becomes res2, and the keyword grad becomes res1
source
Internals
This is not part of the public API.

+df = DataFrame(rows)

The resulting DataFrame will have one column for each of the following fields.

Fields

  • backend::ADTypes.AbstractADType: backend used for benchmarking

  • scenario::Scenario: scenario used for benchmarking

  • operator::Symbol: differentiation operator used for benchmarking, e.g. :gradient or :hessian

  • calls::Int64: number of calls to the differentiated function for one call to the operator

  • samples::Int64: number of benchmarking samples taken

  • evals::Int64: number of evaluations used for averaging in each sample

  • time::Float64: minimum runtime over all samples, in seconds

  • allocs::Float64: minimum number of allocations over all samples

  • bytes::Float64: minimum memory allocated over all samples, in bytes

  • gc_fraction::Float64: minimum fraction of time spent in garbage collection over all samples, between 0.0 and 1.0

  • compile_fraction::Float64: minimum fraction of time spent compiling over all samples, between 0.0 and 1.0

See the documentation of Chairmarks.jl for more details on the measurement fields.

source

Pre-made scenario lists

Scenario types

DifferentiationInterfaceTest.Scenario — Type
Scenario{op,args,pl}

Store a testing scenario composed of a function and its input + output for a given operator.

This generic type should never be used directly: use the specific constructor corresponding to the operator you want to test, or a predefined list of scenarios.

Constructors

Type parameters

  • op: one of :pushforward, :pullback, :derivative, :gradient, :jacobian,:second_derivative, :hvp, :hessian
  • args: either 1 (for f(x) = y) or 2 (for f!(y, x) = nothing)
  • pl: either :inplace or :outofplace

Fields

  • f::Any: function f (if args==1) or f! (if args==2) to apply

  • x::Any: primal input

  • y::Any: primal output

  • seed::Any: seed for pushforward, pullback or HVP

  • res1::Any: first-order result of the operator

  • res2::Any: second-order result of the operator (when it makes sense)

Note that the res1 and res2 fields are given more meaningful names in the keyword arguments of each specialized constructor. For example:

  • the keyword grad of GradientScenario becomes res1
  • the keyword hess of HessianScenario becomes res2, and the keyword grad becomes res1
source

Internals

This is not part of the public API.

diff --git a/DifferentiationInterfaceTest/dev/index.html b/DifferentiationInterfaceTest/dev/index.html index fbeed3636..c1deab410 100644 --- a/DifferentiationInterfaceTest/dev/index.html +++ b/DifferentiationInterfaceTest/dev/index.html @@ -11,4 +11,4 @@ Pkg.add( url="https://github.com/gdalle/DifferentiationInterface.jl", subdir="DifferentiationInterfaceTest" -) +) diff --git a/DifferentiationInterfaceTest/dev/tutorial/index.html b/DifferentiationInterfaceTest/dev/tutorial/index.html index 1af0f5cb6..c4583d99f 100644 --- a/DifferentiationInterfaceTest/dev/tutorial/index.html +++ b/DifferentiationInterfaceTest/dev/tutorial/index.html @@ -13,12 +13,12 @@ type_stability=false, # checks type stability with JET.jl detailed=true, # prints a detailed test set )Test Summary: | Pass Total Time -Testing correctness | 68 68 38.4s +Testing correctness | 68 68 37.8s AutoForwardDiff() | 34 34 4.5s - gradient | 34 34 4.5s - Scenario{:gradient,1,:inplace} f : Vector{Float32} -> Float32 | 17 17 2.3s + gradient | 34 34 4.4s + Scenario{:gradient,1,:inplace} f : Vector{Float32} -> Float32 | 17 17 2.2s Scenario{:gradient,1,:inplace} f : Matrix{Float64} -> Float64 | 17 17 2.2s - AutoEnzyme(mode=EnzymeCore.ReverseMode{false, EnzymeCore.FFIABI, false}()) | 34 34 33.8s - gradient | 34 34 33.8s - Scenario{:gradient,1,:inplace} f : Vector{Float32} -> Float32 | 17 17 31.7s - Scenario{:gradient,1,:inplace} f : Matrix{Float64} -> Float64 | 17 17 2.1s

If you are too lazy to manually specify the reference, you can also provide an AD backend as the ref_backend keyword argument, which will serve as the ground truth for comparison.

Benchmarking

Once you are confident that your backends give the correct answers, you probably want to compare their performance. This is made easy by the benchmark_differentiation function, whose syntax should feel familiar:

df = benchmark_differentiation(backends, scenarios);
12×11 DataFrame
Rowbackendscenariooperatorcallssamplesevalstimeallocsbytesgc_fractioncompile_fraction
Abstract…Scenario…SymbolInt64Int64Int64Float64Float64Float64Float64Float64
1AutoForwardDiff()Scenario{:gradient,1,:inplace} f : Vector{Float32} -> Float32prepare_gradient0111.2563e-511.0528.00.00.0
2AutoForwardDiff()Scenario{:gradient,1,:inplace} f : Vector{Float32} -> Float32value_and_gradient!12815818.0e-81.032.00.00.0
3AutoForwardDiff()Scenario{:gradient,1,:inplace} f : Vector{Float32} -> Float32gradient!13056516.9e-80.00.00.00.0
4AutoForwardDiff()Scenario{:gradient,1,:inplace} f : Matrix{Float64} -> Float64prepare_gradient0111.1191e-511.01776.00.00.0
5AutoForwardDiff()Scenario{:gradient,1,:inplace} f : Matrix{Float64} -> Float64value_and_gradient!12490812.1e-75.0192.00.00.0
6AutoForwardDiff()Scenario{:gradient,1,:inplace} f : Matrix{Float64} -> Float64gradient!12297412.01e-74.0160.00.00.0
7AutoEnzyme(mode=ReverseMode{false, FFIABI, false}())Scenario{:gradient,1,:inplace} f : Vector{Float32} -> Float32prepare_gradient0113.1e-70.00.00.00.0
8AutoEnzyme(mode=ReverseMode{false, FFIABI, false}())Scenario{:gradient,1,:inplace} f : Vector{Float32} -> Float32value_and_gradient!17324816.51e-79.0192.00.00.0
9AutoEnzyme(mode=ReverseMode{false, FFIABI, false}())Scenario{:gradient,1,:inplace} f : Vector{Float32} -> Float32gradient!112795616.9e-80.00.00.00.0
10AutoEnzyme(mode=ReverseMode{false, FFIABI, false}())Scenario{:gradient,1,:inplace} f : Matrix{Float64} -> Float64prepare_gradient0111.4e-70.00.00.00.0
11AutoEnzyme(mode=ReverseMode{false, FFIABI, false}())Scenario{:gradient,1,:inplace} f : Matrix{Float64} -> Float64value_and_gradient!16976717.31e-79.0192.00.00.0
12AutoEnzyme(mode=ReverseMode{false, FFIABI, false}())Scenario{:gradient,1,:inplace} f : Matrix{Float64} -> Float64gradient!112873511.0e-70.00.00.00.0

The resulting object is a DataFrame from DataFrames.jl, whose columns correspond to the fields of DifferentiationBenchmarkDataRow:

+ AutoEnzyme(mode=EnzymeCore.ReverseMode{false, EnzymeCore.FFIABI, false}()) | 34 34 33.3s + gradient | 34 34 33.3s + Scenario{:gradient,1,:inplace} f : Vector{Float32} -> Float32 | 17 17 31.3s + Scenario{:gradient,1,:inplace} f : Matrix{Float64} -> Float64 | 17 17 2.0s

If you are too lazy to manually specify the reference, you can also provide an AD backend as the ref_backend keyword argument, which will serve as the ground truth for comparison.

Benchmarking

Once you are confident that your backends give the correct answers, you probably want to compare their performance. This is made easy by the benchmark_differentiation function, whose syntax should feel familiar:

df = benchmark_differentiation(backends, scenarios);
12×11 DataFrame
Rowbackendscenariooperatorcallssamplesevalstimeallocsbytesgc_fractioncompile_fraction
Abstract…Scenario…SymbolInt64Int64Int64Float64Float64Float64Float64Float64
1AutoForwardDiff()Scenario{:gradient,1,:inplace} f : Vector{Float32} -> Float32prepare_gradient0114.498e-611.0528.00.00.0
2AutoForwardDiff()Scenario{:gradient,1,:inplace} f : Vector{Float32} -> Float32value_and_gradient!13672018.0e-81.032.00.00.0
3AutoForwardDiff()Scenario{:gradient,1,:inplace} f : Vector{Float32} -> Float32gradient!14494516.9e-80.00.00.00.0
4AutoForwardDiff()Scenario{:gradient,1,:inplace} f : Matrix{Float64} -> Float64prepare_gradient0114.499e-611.01776.00.00.0
5AutoForwardDiff()Scenario{:gradient,1,:inplace} f : Matrix{Float64} -> Float64value_and_gradient!13440911.91e-75.0192.00.00.0
6AutoForwardDiff()Scenario{:gradient,1,:inplace} f : Matrix{Float64} -> Float64gradient!12874212.0e-74.0160.00.00.0
7AutoEnzyme(mode=ReverseMode{false, FFIABI, false}())Scenario{:gradient,1,:inplace} f : Vector{Float32} -> Float32prepare_gradient0111.3e-70.00.00.00.0
8AutoEnzyme(mode=ReverseMode{false, FFIABI, false}())Scenario{:gradient,1,:inplace} f : Vector{Float32} -> Float32value_and_gradient!13903018.11e-79.0192.00.00.0
9AutoEnzyme(mode=ReverseMode{false, FFIABI, false}())Scenario{:gradient,1,:inplace} f : Vector{Float32} -> Float32gradient!15947116.9e-80.00.00.00.0
10AutoEnzyme(mode=ReverseMode{false, FFIABI, false}())Scenario{:gradient,1,:inplace} f : Matrix{Float64} -> Float64prepare_gradient0111.4e-70.00.00.00.0
11AutoEnzyme(mode=ReverseMode{false, FFIABI, false}())Scenario{:gradient,1,:inplace} f : Matrix{Float64} -> Float64value_and_gradient!14818518.91e-79.0192.00.00.0
12AutoEnzyme(mode=ReverseMode{false, FFIABI, false}())Scenario{:gradient,1,:inplace} f : Matrix{Float64} -> Float64gradient!113611618.9e-80.00.00.00.0

The resulting object is a DataFrame from DataFrames.jl, whose columns correspond to the fields of DifferentiationBenchmarkDataRow: