From 673c6e5aaf1e0de5b5d0772fc49f7f1cb62845c1 Mon Sep 17 00:00:00 2001 From: Adrian Hill Date: Tue, 20 Aug 2024 18:13:35 +0200 Subject: [PATCH] Document limitations (#175) --- docs/make.jl | 3 +- docs/src/dev/how_it_works.md | 4 +- docs/src/user/api.md | 6 +- docs/src/user/limitations.md | 126 +++++++++++++++++++++++++++++++++++ src/adtypes.jl | 35 ++++++++-- 5 files changed, 160 insertions(+), 14 deletions(-) create mode 100644 docs/src/user/limitations.md diff --git a/docs/make.jl b/docs/make.jl index 8ada095..1ecc9e3 100644 --- a/docs/make.jl +++ b/docs/make.jl @@ -23,7 +23,8 @@ makedocs(; ), pages=[ "Getting Started" => "index.md", - "User Documentation" => ["API Reference" => "user/api.md"], + "User Documentation" => + ["Limitations" => "user/limitations.md", "API Reference" => "user/api.md"], "Developer Documentation" => [ "How SCT works" => "dev/how_it_works.md", "Internals Reference" => "dev/api.md", diff --git a/docs/src/dev/how_it_works.md b/docs/src/dev/how_it_works.md index e921a5a..38e909b 100644 --- a/docs/src/dev/how_it_works.md +++ b/docs/src/dev/how_it_works.md @@ -75,10 +75,10 @@ flowchart LR ``` To obtain a sparsity pattern, each scalar input $x_i$ gets seeded with a corresponding singleton index set $\{i\}$ [^1]. Since addition and multiplication have non-zero derivatives with respect to both of their inputs, -the resulting values accumulate and propagate their index sets (annotated on the edges of the graph above). +their outputs accumulate and propagate the index sets of their inputs (annotated on the edges of the graph above). The sign function has zero derivatives for any input value. It therefore doesn't propagate the index set ${4}$ corresponding to the input $x_4$. Instead, it returns an empty set. -[^1]: since $\frac{\partial x_i}{\partial x_j} \neq 0$ iff $i \neq j$ +[^1]: $\frac{\partial x_i}{\partial x_j} \neq 0$ only holds for $i=j$ The resulting **global** gradient sparsity pattern $\left(\nabla f(\mathbf{x})\right)_{i} \neq 1$ for $i$ in $\{1, 2, 3\}$ matches the analytical gradient diff --git a/docs/src/user/api.md b/docs/src/user/api.md index 37f0056..a9d8447 100644 --- a/docs/src/user/api.md +++ b/docs/src/user/api.md @@ -5,12 +5,8 @@ CollapsedDocStrings = true ``` # [API Reference](@id api) -```@index -``` - -## ADTypes Interface -SparseConnectivityTracer uses [ADTypes.jl](https://github.com/SciML/ADTypes.jl)'s interface for [sparsity detection](https://sciml.github.io/ADTypes.jl/stable/#Sparsity-detector). +SparseConnectivityTracer uses [ADTypes.jl](https://github.com/SciML/ADTypes.jl)'s [interface for sparsity detection](https://sciml.github.io/ADTypes.jl/stable/#Sparsity-detector). In fact, the functions `jacobian_sparsity` and `hessian_sparsity` are re-exported from ADTypes. To compute **global** sparsity patterns of `f(x)` over the entire input domain `x`, use diff --git a/docs/src/user/limitations.md b/docs/src/user/limitations.md new file mode 100644 index 0000000..0ce3662 --- /dev/null +++ b/docs/src/user/limitations.md @@ -0,0 +1,126 @@ +# [Limitations](@id limitations) + +## Sparsity patterns are conservative approximations + +Sparsity patterns returned by SparseConnectivityTracer (SCT) can in some cases be overly conservative, meaning that they might contain "too many ones". +If you observe an overly conservative pattern, [please open a feature request](https://github.com/adrhill/SparseConnectivityTracer.jl/issues) so we know where to add more method overloads to increase the sparsity. + +!!! warning "SCT's no-false-negatives policy" + If you ever observe a sparsity pattern that contains too many zeros, we urge you to [open a bug report](https://github.com/adrhill/SparseConnectivityTracer.jl/issues)! + +## Function must be composed of generic Julia functions + +SCT can't trace through non-Julia code. +However, if you know the sparsity pattern of an external, non-Julia function, +you might be able to work around it by adding methods on SCT's tracer types. + +## Function types must be generic + +When computing the sparsity pattern of a function, +it must be written generically enough to accept numbers of type `T<:Real` as (or `AbstractArray{<:Real}`) as inputs. + +!!! details "Example: Overly restrictive type annotations" + Let's see this mistake in action: + + ```@example notgeneric + using SparseConnectivityTracer + method = TracerSparsityDetector() + + relu_bad(x::AbstractFloat) = max(zero(x), x) + outer_function_bad(xs) = sum(relu_bad, xs) + nothing # hide + ``` + + Since tracers and dual numbers are `Real` numbers and not `AbstractFloat`s, + `relu_bad` throws a `MethodError`: + + ```@repl notgeneric + xs = [1.0, -2.0, 3.0]; + + outer_function_bad(xs) + + jacobian_sparsity(outer_function_bad, xs, method) + ``` + + This is easily fixed by loosening type restrictions or adding an additional methods on `Real`: + + ```@example notgeneric + relu_good(x) = max(zero(x), x) + outer_function_good(xs) = sum(relu_good, xs) + nothing # hide + ``` + + ```@repl notgeneric + jacobian_sparsity(outer_function_good, xs, method) + ``` + +## Limited control flow + +Only [`TracerLocalSparsityDetector`](@ref) supports comparison operators (`<`, `==`, ...), indicator functions (`iszero`, `iseven`, ...) and control flow. + +[`TracerSparsityDetector`](@ref) does not support any boolean functions and control flow (with the exception of `iselse`). +This might seem unintuitive but follows from our policy stated above: SCT guarantees conservative sparsity patterns. +Using an approach based on operator-overloading, this means that global sparsity detection isn't allowed to hit any branching code. +`ifelse` is the only exception, since it allows us to evaluate both branches. + + +!!! warning "Common control flow errors" + By design, SCT will throw errors instead of returning wrong sparsity patterns. Common error messages include: + + ```julia + ERROR: TypeError: non-boolean [tracer type] used in boolean context + ``` + + ```julia + ERROR: Function [function] requires primal value(s). + A dual-number tracer for local sparsity detection can be used via `TracerLocalSparsityDetector`. + ``` + +!!! details "Why does TracerSparsityDetector not support control flow and comparisons?" + Let us motivate the design decision above by a simple example function: + + ```@example ctrlflow + function f(x) + if x[1] > x[2] + return x[1] + else + return x[2] + end + end + nothing # hide + ``` + + The desired **global** Jacobian sparsity pattern over the entire input domain $x \in \mathbb{R}^2$ is `[1 1]`. + Two **local** sparsity patterns are possible: + `[1 0]` for $\{x | x_1 > x_2\}$, + `[0 1]` for $\{x | x_1 \le x_2\}$. + + The local sparsity patterns of [`TracerLocalSparsityDetector`](@ref) are easy to compute using operator overloading by using [dual numbers](@ref SparseConnectivityTracer.Dual) + which contain primal values on which we can evaluate comparisons like `>`: + + ```@repl ctrlflow + using SparseConnectivityTracer + + jacobian_sparsity(f, [2, 1], TracerLocalSparsityDetector()) + + jacobian_sparsity(f, [1, 2], TracerLocalSparsityDetector()) + ``` + + The global sparsity pattern is **impossible** to compute when code branches with an if-else condition, + since we can only ever hit one branch during run-time. + If we made comparisons like `>` return `true` or `false`, we'd get the local patterns `[1 0]` and `[0 1]` respectively. + But SCT's policy is to guarantee conservative sparsity patterns, which means that "false positives" (ones) are acceptable, but "false negatives" (zeros) are not. + In my our opinion, the right thing to do here is to throw an error: + + ```@repl ctrlflow + jacobian_sparsity(f, [1, 2], TracerSparsityDetector()) + ``` + + In some cases, we can work around this by using `ifelse`. + Since `ifelse` is a method, it can evaluate "both branches" and take a conservative union of both resulting sparsity patterns: + + ```@repl ctrlflow + f(x) = ifelse(x[1] > x[2], x[1], x[2]) + + jacobian_sparsity(f, [1, 2], TracerSparsityDetector()) + ``` diff --git a/src/adtypes.jl b/src/adtypes.jl index 6442db1..3453c5c 100644 --- a/src/adtypes.jl +++ b/src/adtypes.jl @@ -19,8 +19,6 @@ julia> jacobian_sparsity(diff, rand(4), TracerSparsityDetector()) ``` ```jldoctest -julia> using SparseConnectivityTracer - julia> f(x) = x[1] + x[2]*x[3] + 1/x[4]; julia> hessian_sparsity(f, rand(4), TracerSparsityDetector()) @@ -67,18 +65,45 @@ For global sparsity patterns, use [`TracerSparsityDetector`](@ref). # Example +Local sparsity patterns are less convervative than global patterns and need to be recomputed for each input `x`: + ```jldoctest julia> using SparseConnectivityTracer +julia> method = TracerLocalSparsityDetector(); + +julia> f(x) = x[1] * x[2]; # J_f = [x[2], x[1]] + +julia> jacobian_sparsity(f, [1, 0], method) +1×2 SparseArrays.SparseMatrixCSC{Bool, Int64} with 1 stored entry: + ⋅ 1 + +julia> jacobian_sparsity(f, [0, 1], method) +1×2 SparseArrays.SparseMatrixCSC{Bool, Int64} with 1 stored entry: + 1 ⋅ + +julia> jacobian_sparsity(f, [0, 0], method) +1×2 SparseArrays.SparseMatrixCSC{Bool, Int64} with 0 stored entries: + ⋅ ⋅ + +julia> jacobian_sparsity(f, [1, 1], method) +1×2 SparseArrays.SparseMatrixCSC{Bool, Int64} with 2 stored entries: + 1 1 +``` + +`TracerLocalSparsityDetector` can compute sparsity patterns of functions that contain comparisons and `ifelse` statements: + + +```jldoctest julia> f(x) = x[1] > x[2] ? x[1:3] : x[2:4]; -julia> jacobian_sparsity(f, [1.0, 2.0, 3.0, 4.0], TracerLocalSparsityDetector()) +julia> jacobian_sparsity(f, [1, 2, 3, 4], TracerLocalSparsityDetector()) 3×4 SparseArrays.SparseMatrixCSC{Bool, Int64} with 3 stored entries: ⋅ 1 ⋅ ⋅ ⋅ ⋅ 1 ⋅ ⋅ ⋅ ⋅ 1 -julia> jacobian_sparsity(f, [2.0, 1.0, 3.0, 4.0], TracerLocalSparsityDetector()) +julia> jacobian_sparsity(f, [2, 1, 3, 4], TracerLocalSparsityDetector()) 3×4 SparseArrays.SparseMatrixCSC{Bool, Int64} with 3 stored entries: 1 ⋅ ⋅ ⋅ ⋅ 1 ⋅ ⋅ @@ -86,8 +111,6 @@ julia> jacobian_sparsity(f, [2.0, 1.0, 3.0, 4.0], TracerLocalSparsityDetector()) ``` ```jldoctest -julia> using SparseConnectivityTracer - julia> f(x) = x[1] + max(x[2], x[3]) * x[3] + 1/x[4]; julia> hessian_sparsity(f, [1.0, 2.0, 3.0, 4.0], TracerLocalSparsityDetector())