API reference
DifferentiationInterfaceTest — ModuleDifferentiationInterfaceTestTesting and benchmarking utilities for automatic differentiation in Julia.
Exports
DerivativeScenarioDifferentiationBenchmarkDataRowGradientScenarioHVPScenarioHessianScenarioJacobianScenarioPullbackScenarioPushforwardScenarioScenarioSecondDerivativeScenariobenchmark_differentiationcomponent_scenariosdefault_scenariosgpu_scenariossparse_scenariosstatic_scenariostest_differentiation
Entry points
DifferentiationInterfaceTest.test_differentiation — Functiontest_differentiation(
+ backends::Vector{<:ADTypes.AbstractADType};
+ ...
+)
+test_differentiation(
+ backends::Vector{<:ADTypes.AbstractADType},
+ scenarios::Vector{<:Scenario};
+ correctness,
+ type_stability,
+ call_count,
+ sparsity,
+ detailed,
+ input_type,
+ output_type,
+ first_order,
+ second_order,
+ onearg,
+ twoarg,
+ inplace,
+ outofplace,
+ excluded,
+ logging,
+ isequal,
+ isapprox,
+ atol,
+ rtol
+)
+Cross-test a list of backends on a list of scenarios, running a variety of different tests.
Default arguments
scenarios::Vector{<:Scenario}: the output ofdefault_scenarios()
Keyword arguments
Testing:
correctness=true: whether to compare the differentiation results with the theoretical values specified in each scenariotype_stability=false: whether to check type stability with JET.jl (thanks toJET.@test_opt)sparsity: whether to check sparsity of the jacobian / hessiandetailed=false: whether to print a detailed or condensed test log
Filtering:
input_type=Any,output_type=Any: restrict scenario inputs / outputs to subtypes of thisfirst_order=true,second_order=true: include first order / second order operatorsonearg=true,twoarg=true: include one-argument / two-argument functionsinplace=true,outofplace=true: include in-place / out-of-place operators
Options:
logging=false: whether to log progressisequal=isequal: function used to compare objects exactly, with the standard signatureisequal(x, y)isapprox=isapprox: function used to compare objects approximately, with the standard signatureisapprox(x, y; atol, rtol)atol=0: absolute precision for correctness testing (when comparing to the reference outputs)rtol=1e-3: relative precision for correctness testing (when comparing to the reference outputs)
test_differentiation(
+ backend::ADTypes.AbstractADType,
+ args...;
+ kwargs...
+)
+Shortcut for a single backend.
DifferentiationInterfaceTest.benchmark_differentiation — Functionbenchmark_differentiation(
+ backends::Vector{<:ADTypes.AbstractADType},
+ scenarios::Vector{<:Scenario};
+ input_type,
+ output_type,
+ first_order,
+ second_order,
+ onearg,
+ twoarg,
+ inplace,
+ outofplace,
+ excluded,
+ logging
+) -> DataFrames.DataFrame
+Benchmark a list of backends for a list of operators on a list of scenarios.
The object returned is a DataFrames.DataFrame where each column corresponds to a field of DifferentiationBenchmarkDataRow.
The keyword arguments available here have the same meaning as those in test_differentiation.
DifferentiationInterfaceTest.DifferentiationBenchmarkDataRow — TypeDifferentiationBenchmarkDataRowAd-hoc storage type for differentiation benchmarking results.
If you have a vector rows::Vector{DifferentiationBenchmarkDataRow}, you can turn it into a DataFrame as follows:
using DataFrames
+
+df = DataFrame(rows)The resulting DataFrame will have one column for each of the following fields.
Fields
backend::ADTypes.AbstractADType: backend used for benchmarkingscenario::Scenario: scenario used for benchmarkingoperator::Symbol: differentiation operator used for benchmarking, e.g.:gradientor:hessiancalls::Int64: number of calls to the differentiated function for one call to the operatorsamples::Int64: number of benchmarking samples takenevals::Int64: number of evaluations used for averaging in each sampletime::Float64: minimum runtime over all samples, in secondsallocs::Float64: minimum number of allocations over all samplesbytes::Float64: minimum memory allocated over all samples, in bytesgc_fraction::Float64: minimum fraction of time spent in garbage collection over all samples, between 0.0 and 1.0compile_fraction::Float64: minimum fraction of time spent compiling over all samples, between 0.0 and 1.0
See the documentation of Chairmarks.jl for more details on the measurement fields.
Pre-made scenario lists
The precise contents of the scenario lists are not part of the API, only their existence.
DifferentiationInterfaceTest.default_scenarios — Functiondefault_scenarios(rng=Random.default_rng())Create a vector of Scenarios with standard array types.
DifferentiationInterfaceTest.sparse_scenarios — Functionsparse_scenarios(rng=Random.default_rng())Create a vector of Scenarios with sparse array types, focused on sparse Jacobians and Hessians.
DifferentiationInterfaceTest.component_scenarios — Functioncomponent_scenarios(rng=Random.default_rng())Create a vector of Scenarios with component array types from ComponentArrays.jl.
This function requires ComponentArrays.jl to be loaded (it is implemented in a package extension).
DifferentiationInterfaceTest.gpu_scenarios — Functiongpu_scenarios(rng=Random.default_rng())Create a vector of Scenarios with GPU array types from JLArrays.jl.
This function requires JLArrays.jl to be loaded (it is implemented in a package extension).
DifferentiationInterfaceTest.static_scenarios — Functionstatic_scenarios(rng=Random.default_rng())Create a vector of Scenarios with static array types from StaticArrays.jl.
This function requires StaticArrays.jl to be loaded (it is implemented in a package extension).
Scenario types
DifferentiationInterfaceTest.Scenario — TypeScenario{op,args,pl}Store a testing scenario composed of a function and its input + output for a given operator.
This generic type should never be used directly: use the specific constructor corresponding to the operator you want to test, or a predefined list of scenarios.
Constructors
PushforwardScenarioPullbackScenarioDerivativeScenarioGradientScenarioJacobianScenarioSecondDerivativeScenarioHVPScenarioHessianScenario
Type parameters
op: one of:pushforward,:pullback,:derivative,:gradient,:jacobian,:second_derivative,:hvp,:hessianargs: either1(forf(x) = y) or2(forf!(y, x) = nothing)pl: either:inplaceor:outofplace
Fields
f::Any: functionf(ifargs==1) orf!(ifargs==2) to applyx::Any: primal inputy::Any: primal outputseed::Any: seed for pushforward, pullback or HVPres1::Any: first-order result of the operatorres2::Any: second-order result of the operator (when it makes sense)
Note that the res1 and res2 fields are given more meaningful names in the keyword arguments of each specialized constructor. For example:
- the keyword
gradofGradientScenariobecomesres1 - the keyword
hessofHessianScenariobecomesres2, and the keywordgradbecomesres1
DifferentiationInterfaceTest.PushforwardScenario — FunctionPushforwardScenario(f; x, y, dx, dy, nb_args, place)
+Construct a Scenario to test pushforward and its variants.
DifferentiationInterfaceTest.PullbackScenario — FunctionPullbackScenario(f; x, y, dy, dx, nb_args, place)
+Construct a Scenario to test pullback and its variants.
DifferentiationInterfaceTest.DerivativeScenario — FunctionDerivativeScenario(f; x, y, der, nb_args, place)
+Construct a Scenario to test derivative and its variants.
DifferentiationInterfaceTest.GradientScenario — FunctionGradientScenario(f; x, y, grad, nb_args, place)
+Construct a Scenario to test gradient and its variants.
DifferentiationInterfaceTest.JacobianScenario — FunctionJacobianScenario(f; x, y, jac, nb_args, place)
+Construct a Scenario to test jacobian and its variants.
DifferentiationInterfaceTest.SecondDerivativeScenario — FunctionSecondDerivativeScenario(f; x, y, der, der2, nb_args, place)
+Construct a Scenario to test second_derivative and its variants.
DifferentiationInterfaceTest.HVPScenario — FunctionHVPScenario(f; x, y, dx, grad, dg, nb_args, place)
+Construct a Scenario to test hvp and its variants.
DifferentiationInterfaceTest.HessianScenario — FunctionHessianScenario(f; x, y, grad, hess, nb_args, place)
+Construct a Scenario to test hessian and its variants.
Internals
This is not part of the public API.
DifferentiationInterfaceTest.AutoZeroForward — TypeAutoZeroForward <: ADTypes.AbstractADTypeTrivial backend that sets all derivatives to zero. Used in testing and benchmarking.
DifferentiationInterfaceTest.AutoZeroReverse — TypeAutoZeroReverse <: ADTypes.AbstractADTypeTrivial backend that sets all derivatives to zero. Used in testing and benchmarking.
DifferentiationInterfaceTest.add_batchified! — Methodadd_batchified!(scens::AbstractVector{<:Scenario})Add batchified versions to scens of its scenarios which support it (pushforward, pullback and HVP).
DifferentiationInterfaceTest.allocfree_scenarios — Functionallocfree_scenarios(rng::AbstractRNG=default_rng())Create a vector of Scenarios with functions that do not allocate.
At the moment, some second-order scenarios are excluded.
DifferentiationInterfaceTest.flux_isapprox — Functionflux_isapprox(x, y; atol, rtol)Approximate comparison function to use in correctness tests with gradients of Flux.jl networks.
DifferentiationInterfaceTest.flux_isequal — Functionflux_isequal(x, y)Exact comparison function to use in correctness tests with gradients of Flux.jl networks.
DifferentiationInterfaceTest.flux_scenarios — Functionflux_scenarios(rng=Random.default_rng())Create a vector of Scenarios with neural networks from Flux.jl.
This function requires Flux.jl and FiniteDifferences.jl to be loaded (it is implemented in a package extension).
These scenarios are still experimental and not part of the public API. Their ground truth values are computed with finite differences, and thus subject to imprecision.
DifferentiationInterfaceTest.make_closure — Methodmake_closure(scen::Scenario)Return a new Scenario with a modified function f or f! that closes over differentiable data.
DifferentiationInterfaceTest.test_allocfree — Methodtest_allocfree(benchmark_data::DataFrame)Test that every row in benchmark_data which is not a preparation row has zero allocation.
