Question about ART's Black Box Attack on Tesseract notebook example

[HexHexHex16]

ART Black Box Attack on Tesseract example:
https://github.com/Trusted-AI/adversarial-robustness-toolbox/blob/fd9e54892386a32b32e10e87fe86bfa22be1e3e8/notebooks/classifier_blackbox_tesseract.ipynb

As far as I understand notebook should have worked like this:

1. Import libraries
2. Reading picture 1 (used for overlay) and picture 2 (on which is superimposed)
3. Setting the prediction function to call tesseract from the command line and transform
4. Creating a dictionary with the keys of recognizable words on pictures 1 and 2
5. Word recognition in picture 1
6. Word recognition in picture 2
7. Attack HopSkipJump (overlay picture 1 on 2)
   {Result: Recognition of the word "dissent" in the processed picture}
8. Apply JPEG-COMPRESSION
9. Word recognition in picture 1
10. Word recognition in picture 2
11. Attack HopSkipJump (overlay picture 1 on 2)
    {Result: Recognition of the word "assent" in the processed picture}

But, the problem is that in 7) the processed "assent image " is recognized as "assent" and not as "dissent" during the attack, although the processed image looks like "dissent".
To work correctly, the result on 7) must be "dissent" after the attack, and on 11) "assent", becasue in 8) the JPEG-COMPRESSION preprocessing method will be used

[beat-buesser]

Hi @HexHexHex16 I think you are right, cell 7 should run for more iterations until Tesseract predicts dissent. Cell 11 will also eventually, with more iterations show dissent, but the notebook aimed to show that a larger perturbation is required.

Let me know if you are updating the notebook, I'd be happy to pull it into the main branch.

[beat-buesser]

@HexHexHex16 Are you able to extract the file out.txt created by Tesseract and show its content? It should contain the current prediction by Tesseract but it seems that it now contains a character that Python doesn't expect. This could indicate that the prediction has changed but that we need to adapt the character map used to read the file.

[HexHexHex16]

Yes, here is the contents of the file out.txt ( ‘dissent )

[beat-buesser]

I think to continue iterating the attack to reduce the perturbation further we would have to set the correct endoding in the call to open in cell 3. The content of out.txt also shows that the perturbation of the last iteration isn't able to fool Tesseract to return "assent" anymore, but " 'dissent" which is closer to the "dissent" shown in the image, therefore less adversarial.

[HexHexHex16]

How exactly can I eliminate this character in ['dissent] by modifying the open() method in cell 3?

[beat-buesser]

Maybe a different encoding in the call to open might help.