Regarding code-friendly EoT attacks

[Lodour]

Existing EoT attacks are implemented on the preprocessor part by subclassing EoTPyTorch (or EoTTensorFlowV2).

I encounter several drawbacks when dealing with a large number of preprocessors:

1. I have to write an EoT class for every existing preprocessor (potentially for every framework in ART's own case).
2. I have to manually decrease the batch size due to memory constraints, as EoTPyTorch works by duplicating the inputs.
3. EoTPyTorch is not optimized for multiple randomized preprocessors (e.g., duplicated samples are duplicated again in the next eot preprocessor, leading to exponential sizes). I have to write an "ensemble preprocessor" to manually forward on multiple preprocessors.
4. I need a non-EoT estimator to do the prediction (to avoid test-time EoT).

On my side, I found a better solution for EoT by wrapping the estimator's loss_gradient and class_gradient methods, so they repeat themselves multiple times and average the returned gradients. In the meanwhile, the predict method works as if without EoT.

The code below works well on my side and addresses all drawbacks I encountered. But it looks hacky to the ART framework, and I am not sure if (and how) it properly fits into the framework without affecting other features that I am not aware of. I am willing to contribute this feature if you find it useful for ART.

Thanks!

import functools
import types
from typing import Callable

from art.estimators.classification import PyTorchClassifier


def averaged_method(method: Callable, n_calls: int):
    """
    Make a method average its outputs over multiple calls.

    :param method: Bounded method of an instance.
    :param n_calls: Number of calls.
    :return: Averaged outputs.
    """

    assert isinstance(method, types.MethodType), f'Expected method type, but got {type(method)}.'
    assert n_calls >= 1, f'n_calls must be positive, but got {n_calls}.'

    @functools.wraps(method)
    def wrapper(self, *args, **kwargs):
        outputs = method(*args, **kwargs)  # this is invariant to the method's framework
        for _ in range(n_calls - 1):
            outputs += method(*args, **kwargs)  # this operation does not increase too much memory
        return outputs / n_calls

    return types.MethodType(wrapper, method.__self__)


if __name__ == '__main__':
    classifier = PyTorchClassifier(..., preprocessing_defences=[randomized_defense_without_eot])
    classifier.loss_gradient = averaged_method(classifier.loss_gradient, n_calls=10)
    classifier.class_gradient = averaged_method(classifier.class_gradient, n_calls=10)

[beat-buesser]

Hi @Lodour Thank you very much for your great proposal! You are right that the framework-specific EoT preprocessors can reach memory limitations if the number of EoT samples is too large for the available memory. I think your observations are correct, but let me add a few extensions:

I have to write an EoT class for every existing preprocessor (potentially for every framework in ART's own case).

In the new class you only should have to implement the _transform method along with checks for required arguments. A reason for the current

I have to manually decrease the batch size due to memory constraints, as EoTPyTorch works by duplicating the inputs.

Yes, that might be necessary. The approach of duplication of the inputs focuses on speed to allow all EoT samples to be calculated at the same time, but is sooner limited by memory.

EoTPyTorch is not optimized for multiple randomized preprocessors (e.g., duplicated samples are duplicated again in the next eot preprocessor, leading to exponential sizes). I have to write an "ensemble preprocessor" to manually forward on multiple preprocessors.

To prevent an increase in the number of EoT samples with multiple EoT preprocessors in sequence it is possible to define the number of samples in the first EoT prerpocessor and keep the number of samples in the subsequent EoT preprocessors at 1, this will avoid further expansion of the number of samples.

I need a non-EoT estimator to do the prediction (to avoid test-time EoT).

The arguments apply_fit and apply_predict of each preprocessor allow defining if a preprocessor is applied during calls to method fit or predict to allow using the same estimator for evaluation, etc.

I think the current code above would be likely a Numpy based preprocessor. Would it be possible to implement it framework-specific e.g. in PyTorch to allow gradients flow further backwards from the outputs of the averaging step?

I'm wondering if the averaging step to counter randomness is really best fit inside an EoT preprocessors or if it is better a property of how the estimators apply the preprocessors. I'm thinking if it could be an option to method _apply_preprocessing:

adversarial-robustness-toolbox/art/estimators/pytorch.py

Line 137 in 3cf9b36

def _apply_preprocessing(self, x, y, fit: bool = False, no_grad=True) -> Tuple[Any, Any]: # pylint: disable=W0221

or in the method forward of the EoT preprocessor base classes:

adversarial-robustness-toolbox/art/preprocessing/expectation_over_transformation/pytorch.py

Line 72 in 3cf9b36

def forward(

But both still raise the question if we can backpropagate gradients in the framework's tensors when averaging in a looped samples instead of in parallel samples.

I am willing to contribute this feature if you find it useful for ART.

That's great, let's continue the discussion to find a best place for this interesting feature, I'm sure we can make it work and include it in a future release.

[Lodour]

Thanks for your reply!

The extensions you clarified are very useful, I almost forgot them.

I would like first to clarify some of my comments:

In the new class you only should have to implement the _transform method along with checks for required arguments.

I personally prefer maintaining a group of defense classes (without EoT) and enabling EoT only when necessary (without creating a new class and wrapping the defense in PyTorch and TF's _transform). I guess what I am trying to do is move the overheads of coding from the preprocessor's side to the classifier's side, because making the classifier duplicate inputs in a preprocessor-agnostic way seems to have less code.

The approach of duplication of the inputs focuses on speed to allow all EoT samples to be calculated at the same time, but is sooner limited by memory.

In my use case, the memory is already fully occupied with a batch size of B (before applying EoT). I guess in this case we have to separate the computation into several passes anyway. Say the EoT sample is N, it should be one of the following:

1. The entire batch B computed for N times (as in my proposal).
2. The entire batch B split into N mini-batches of sizes B/N and each mini-batch computed separately (as in the current solution).

These two cases should have similar computation "times".

One exception is that BN is acceptable by the memory, in this case, the current duplication should be fine. Not yet sure how to make this acceptable BN case and the above largest B case both happy.

I need a non-EoT estimator to do the prediction (to avoid test-time EoT).

Apologize for the confusion, I meant to avoid "predict-time EoT" as mentioned in this notebook's Step 7: "The evaluation was done with predictions of the ART classifier without EoT, but the same classifier model, created first above, to not add additional rotation to the evaluated input."

Backward from the outputs of the averaging step:

Would it be possible to implement it framework-specific e.g. in PyTorch to allow gradients flow further backwards from the outputs of the averaging step?

Current loss_gradient always detaches the input tensor, so the gradient will not flow back further.

adversarial-robustness-toolbox/art/estimators/classification/pytorch.py

Lines 757 to 768 in 3cf9b36

	# Apply preprocessing
	if self.all_framework_preprocessing:
	if isinstance(x, torch.Tensor):
	x_grad = x.clone().detach().requires_grad_(True)
	else:
	x_grad = torch.tensor(x).to(self._device)
	x_grad.requires_grad = True
	if isinstance(y, torch.Tensor):
	y_grad = y.clone().detach()
	else:
	y_grad = torch.tensor(y).to(self._device)
	inputs_t, y_preprocessed = self._apply_preprocessing(x_grad, y=y_grad, fit=False, no_grad=False)

But if you meant for the PyTorch tensor graph, I think the gradients can flow back further:

import torch


def preprocess(inputs: torch.Tensor):
    return inputs * 2.0


def averaged(inputs: torch.Tensor, n_calls: int):
    outputs = preprocess(inputs)
    for _ in range(n_calls - 1):
        outputs += preprocess(inputs)
    return outputs / n_calls


# simple network
x = torch.ones(1).requires_grad_(True)
y = x * 10
y.retain_grad()

# backward on averaged runs
loss = averaged(y, n_calls=10)
loss.backward()

# grad w.r.t. direct input
print(y.grad)  # tensor([2.])

# grad w.r.t. leaf node
print(x.grad)  # tensor([20.])

Where to put the averaging step:

I'm wondering if the averaging step to counter randomness is really best fit inside an EoT preprocessors or if it is better a property of how the estimators apply the preprocessor.

I am not super clear what is the best choice. The estimator's _apply_preprocessing could be a choice, but it is correlated with too many other components. The best place I could think of right now is to let the attacker determine the estimator's behavior. For example, the attacker can choose to attack with or without EoT, and modify the estimator accordingly.

[beat-buesser]

Hi @Lodour Thank you for explanations.

Is your main goal to average out randomness in preprocessing/defense steps?

The current architecture of preprocessing.expectation_over_transformation was also motivated to support sampling over EoT parameters like rotation angles from ranges or discrete values and to support different types of labels their transformation.

Yes, loss_gradient cannot be connected into a graph. The requirement for gradient backporpagation through EoTs comes for example from the method predict which can be used on tensors to get tensors as output to allow attacks to connect the preprocessing and model into a graph and extended it, e.g. with optimisers, custom losses, etc. which is very useful for example for patch evasion or poisoning attacks.

What do you think if we create an implementation of EoTPyTorch for averaging e.g. EoTPyTorchAverage which takes as input an instance of an implementation of PreprocessorPyTorch to automatically support all predecessors? EoTPyTorchAverage could then define how many evaluations of the provided preprocessor to average, etc.