Improving hash library workflow

[bjlittle]

For the current workflow, if there are hash library test failures for a pytest session, then these are easily discoverable thanks to the rich loveliness of the html figure comparison.

However, given any single hash library failure, the outcome is essentially three-fold:

1. the graphic test has detected a genuine failure that requires to be resolved (somehow)
2. the tolerance of the test requires to be increased
3. a new hash value requires to be associated with the test and recorded in the hash library along with a new image

The choice of including or not including a new hash can result in a blend of the existing hash library and a new hash library (as opposed to accepting all new hashes).

We could extend the workflow to offer the user the choice to manually accept or reject any new proposed hashes. This would ease the burden of customising the hash library and also resolving any result-failed-diff.png images, all of which should then be placed under version control (somewhere).

We adopt this approach on SciTools/iris and SciTools/tephi, for example see SciTools/tephi#78.

[Cadair]

I am not sure I am 100% following what you are saying here.

I don't think there should ever be the case where a hash comparison fails. i.e. you shouldn't be using the tolerance parameter in the image test to "bypass" a hash fail. So I am not entirely sure why you would ever want to not accept all the changes to a hash library.

(I am not sure that you can't do what I think you are suggesting, I just don't think you should)

[rcomer]

Matplotlib has triage_tests.py, which I think is a similar idea (though they are accepting/rejecting new images rather than hashes).