ZOOAttack Sucess rate

[maithal]

Hi,

I am trying to implement a ZOO Attack using the ART toolbox and I am using targeted attack settings. However, the ASR is always 0%. I looked in prior issues and tried changing the model to output logits but still, the problem persists.

attack = ZooAttack(classifier=classifier, batch_size=1, targeted=True, learning_rate=1e-3,
max_iter=3000, binary_search_steps=9,initial_const=10)

Is there something I am missing? Any help will be greatly appreciated!

[beat-buesser]

Hi @maithal Thank you very much for using ART! Could you please describe the dataset and how you define classifier?

[maithal]

Hi,

Thanks for the quick response!

I am trying to use the MNIST dataset and the model I used is defined as follows:

class TensorFlowModel(Model):
def init(self):
super(TensorFlowModel, self).init()
self.conv1 = Conv2D(filters=4, kernel_size=5, activation="relu")
self.conv2 = Conv2D(filters=10, kernel_size=5, activation="relu")
self.maxpool = MaxPool2D(pool_size=(2, 2), strides=(2, 2), padding="valid", data_format=None)
self.flatten = Flatten()
self.dense1 = Dense(100, activation="relu")
self.logits = Dense(10, activation="linear")

def call(self, x):
    """
    Call function to evaluate the model.

    :param x: Input to the model
    :return: Prediction of the model
    """
    x = self.conv1(x)
    x = self.maxpool(x)
    x = self.conv2(x)
    x = self.maxpool(x)
    x = self.flatten(x)
    x = self.dense1(x)
    x = self.logits(x)
    return x

optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)
loss_object = tf.keras.losses.CategoricalCrossentropy(from_logits=True)

Please let me know if there is an issue with the classifier.

[beat-buesser]

I think the model definition looks good. How do you define classifier = ...?

[maithal]

I defined the classifier as shown below:

classifier = TensorFlowV2Classifier(
model=model,
loss_object=loss_object,
train_step=train_step,
nb_classes=10,
input_shape=(28, 28, 1),
clip_values=(0, 1),
)

[beat-buesser]

Ok, that looks good. After training your model on the MNIST training set, what is the training and test classification accuracy?

[maithal]

The model accuracy was 100% after training. Is it related to the input parameters of Zoo Attack?

[beat-buesser]

@maithal Thank you for the link. The notebook has currently restricted access, can you make it public or grant access?

[maithal]

Hi, sorry for the inconvenience! I changed the access to the public now. Please let me know if there is any issue.

[beat-buesser]

I think the attack should be working, but it looks like it overwrites successful adversarial examples in subsequent binary search steps if the distance is smaller despite that the predicted labels changes. We should consider adding a check to prevent this. It could also happen more frequently because of the dataset MNIST and its properties. Could you please try the notebook with CIFAR-10? I have saved my changes to the notebook.

[maithal]

The ASR changes to 100% for the untargeted case but if I use the targeted attack, the ASR is 0%. Can you let me know what is the reason the targeted attack is failing?

[beat-buesser]

That's great news! Can you share how you set up the targeted attack? The access to the notebook is restricted again.