Basic API
All methods in ExplainableAI.jl work by calling analyze
on an input and an analyzer:
ExplainableAI.analyze
— Functionanalyze(input, method)
-analyze(input, method, neuron_selection)
Apply the analyzer method
for the given input, returning an Explanation
. If neuron_selection
is specified, the explanation will be calculated for that neuron. Otherwise, the output neuron with the highest activation is automatically chosen.
See also Explanation
and heatmap
.
Keyword arguments
add_batch_dim
: add batch dimension to the input without allocating. Default isfalse
.
ExplainableAI.Explanation
— TypeReturn type of analyzers when calling analyze
.
Fields
val
: numerical output of the analyzer, e.g. an attribution or gradientoutput
: model output for the given analyzer inputneuron_selection
: neuron index used for the explanationanalyzer
: symbol corresponding the used analyzer, e.g.:LRP
or:Gradient
extras
: optional named tuple that can be used by analyzers to return additional information.
ExplainableAI.heatmap
— Functionheatmap(explanation)
+analyze(input, method, neuron_selection)
Apply the analyzer method
for the given input, returning an Explanation
. If neuron_selection
is specified, the explanation will be calculated for that neuron. Otherwise, the output neuron with the highest activation is automatically chosen.
See also Explanation
and heatmap
.
Keyword arguments
add_batch_dim
: add batch dimension to the input without allocating. Default isfalse
.
ExplainableAI.Explanation
— TypeReturn type of analyzers when calling analyze
.
Fields
val
: numerical output of the analyzer, e.g. an attribution or gradientoutput
: model output for the given analyzer inputneuron_selection
: neuron index used for the explanationanalyzer
: symbol corresponding the used analyzer, e.g.:LRP
or:Gradient
extras
: optional named tuple that can be used by analyzers to return additional information.
ExplainableAI.heatmap
— Functionheatmap(explanation)
heatmap(input, analyzer)
-heatmap(input, analyzer, neuron_selection)
Visualize explanation. Assumes Flux's WHCN convention (width, height, color channels, batch size).
See also analyze
.
Keyword arguments
cs::ColorScheme
: color scheme from ColorSchemes.jl that is applied. When callingheatmap
with anExplanation
or analyzer, the method default is selected. When callingheatmap
with an array, the default isColorSchemes.seismic
.reduce::Symbol
: selects how color channels are reduced to a single number to apply a color scheme. The following methods can be selected, which are then applied over the color channels for each "pixel" in the explanation::sum
: sum up color channels:norm
: compute 2-norm over the color channels:maxabs
: computemaximum(abs, x)
over the color channels
heatmap
with anExplanation
or analyzer, the method default is selected. When callingheatmap
with an array, the default is:sum
.rangescale::Symbol
: selects how the color channel reduced heatmap is normalized before the color scheme is applied. Can be either:extrema
or:centered
. When callingheatmap
with anExplanation
or analyzer, the method default is selected. When callingheatmap
with an array, the default for use with theseismic
color scheme is:centered
.permute::Bool
: Whether to flip W&H input channels. Default istrue
.unpack_singleton::Bool
: When heatmapping a batch with a single sample, settingunpack_singleton=true
will return an image instead of an Vector containing a single image.
Note: keyword arguments can't be used when calling heatmap
with an analyzer.
Analyzers
ExplainableAI.LRP
— TypeLRP(model, rules)
-LRP(model, composite)
Analyze model by applying Layer-Wise Relevance Propagation. The analyzer can either be created by passing an array of LRP-rules or by passing a composite, see Composite
for an example.
Keyword arguments
skip_checks::Bool
: Skip checks whether model is compatible with LRP and contains output softmax. Default isfalse
.verbose::Bool
: Select whether the model checks should print a summary on failure. Default istrue
.
References
[1] G. Montavon et al., Layer-Wise Relevance Propagation: An Overview [2] W. Samek et al., Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
ExplainableAI.Gradient
— TypeGradient(model)
Analyze model by calculating the gradient of a neuron activation with respect to the input.
ExplainableAI.InputTimesGradient
— TypeInputTimesGradient(model)
Analyze model by calculating the gradient of a neuron activation with respect to the input. This gradient is then multiplied element-wise with the input.
ExplainableAI.SmoothGrad
— FunctionSmoothGrad(analyzer, [n=50, std=0.1, rng=GLOBAL_RNG])
-SmoothGrad(analyzer, [n=50, distribution=Normal(0, σ²=0.01), rng=GLOBAL_RNG])
Analyze model by calculating a smoothed sensitivity map. This is done by averaging sensitivity maps of a Gradient
analyzer over random samples in a neighborhood of the input, typically by adding Gaussian noise with mean 0.
References
- Smilkov et al., SmoothGrad: removing noise by adding noise
ExplainableAI.IntegratedGradients
— FunctionIntegratedGradients(analyzer, [n=50])
-IntegratedGradients(analyzer, [n=50])
Analyze model by using the Integrated Gradients method.
References
- Sundararajan et al., Axiomatic Attribution for Deep Networks
Input augmentations
SmoothGrad
and IntegratedGradients
are special cases of the input augmentations NoiseAugmentation
and InterpolationAugmentation
, which can be applied as a wrapper to any analyzer:
ExplainableAI.NoiseAugmentation
— TypeNoiseAugmentation(analyzer, n, [std=1, rng=GLOBAL_RNG])
-NoiseAugmentation(analyzer, n, distribution, [rng=GLOBAL_RNG])
A wrapper around analyzers that augments the input with n
samples of additive noise sampled from distribution
. This input augmentation is then averaged to return an Explanation
.
ExplainableAI.InterpolationAugmentation
— TypeInterpolationAugmentation(model, [n=50])
A wrapper around analyzers that augments the input with n
steps of linear interpolation between the input and a reference input (typically zero(input)
). The gradients w.r.t. this augmented input are then averaged and multiplied with the difference between the input and the reference input.
Model preparation
ExplainableAI.strip_softmax
— Functionstrip_softmax(model)
-strip_softmax(layer)
Remove softmax activation on layer or model if it exists.
ExplainableAI.canonize
— Functioncanonize(model)
Canonize model by flattening it and fusing BatchNorm layers into preceding Dense and Conv layers with linear activation functions.
ExplainableAI.flatten_model
— Functionflatten_model(model)
Flatten a Flux Chain
containing Chain
s.
Input preprocessing
ExplainableAI.preprocess_imagenet
— Functionpreprocess_imagenet(img)
Preprocess an image for use with Metalhead.jl's ImageNet models using PyTorch weights. Uses PyTorch's normalization constants.
Index
ExplainableAI.AlphaBetaRule
ExplainableAI.ChainTuple
ExplainableAI.Composite
ExplainableAI.ConvLayer
ExplainableAI.DropoutLayer
ExplainableAI.EpsilonRule
ExplainableAI.Explanation
ExplainableAI.FirstLayerMap
ExplainableAI.FirstLayerTypeMap
ExplainableAI.FirstNTypeMap
ExplainableAI.FlatRule
ExplainableAI.GammaRule
ExplainableAI.GeneralizedGammaRule
ExplainableAI.GlobalMap
ExplainableAI.GlobalTypeMap
ExplainableAI.Gradient
ExplainableAI.InputTimesGradient
ExplainableAI.InterpolationAugmentation
ExplainableAI.LRP
ExplainableAI.LastLayerMap
ExplainableAI.LastLayerTypeMap
ExplainableAI.LayerMap
ExplainableAI.NoiseAugmentation
ExplainableAI.NormalizationLayer
ExplainableAI.ParallelTuple
ExplainableAI.PassRule
ExplainableAI.PoolingLayer
ExplainableAI.RangeMap
ExplainableAI.RangeTypeMap
ExplainableAI.ReshapingLayer
ExplainableAI.WSquareRule
ExplainableAI.ZBoxRule
ExplainableAI.ZPlusRule
ExplainableAI.ZeroRule
ExplainableAI.EpsilonAlpha2Beta1
ExplainableAI.EpsilonAlpha2Beta1Flat
ExplainableAI.EpsilonGammaBox
ExplainableAI.EpsilonPlus
ExplainableAI.EpsilonPlusFlat
ExplainableAI.IntegratedGradients
ExplainableAI.LRP_CONFIG.supports_activation
ExplainableAI.LRP_CONFIG.supports_layer
ExplainableAI.SmoothGrad
ExplainableAI.analyze
ExplainableAI.canonize
ExplainableAI.flatten_model
ExplainableAI.heatmap
ExplainableAI.is_compatible
ExplainableAI.lrp_rules
ExplainableAI.modify_bias
ExplainableAI.modify_denominator
ExplainableAI.modify_input
ExplainableAI.modify_layer
ExplainableAI.modify_parameters
ExplainableAI.modify_weight
ExplainableAI.preprocess_imagenet
ExplainableAI.show_layer_indices
ExplainableAI.strip_softmax