compare screenshots during automated testing

[pixelzoom]

Moved here from 8/17/17 dev meeting notes, since this will involve considerable brainstorming, design, identification of pitfalls, etc.

CM: Perhaps we should increase the priority of comparing screenshots during automated testing. PhET has too many products to manually review them. Post-vacation, I had 5 sims with serious layout problems. That’s a lot of “drift” over the course of only ~10 days.
JO: I agree. Collaboration with SR may be helpful, as I don’t see a non-phet-io way of setting the seed on launch (query parameter?), sending it an event stream or triggering a predefined random fuzz (postmessage?) or getting a screenshot out (postmessage?)
JB: Seems like a great idea if we can do it for a reasonable cost.
JB: Bumping to next dev meeting (Aug 17) so we can continue discussing with AP, JO, and CM, who are out for Aug 10 meeting.

[jonathanolson]

I was most interested in being able to determine "whether things changed in a sim", and ideally "what changed visually" to help tracking it down.

I was thinking that we'd be able to initialize the sim with a random seed, send it fuzz-like events/frames with another (constant) random seed, and take screenshots at predefined intervals. For each sim, it would be a series of screenshots (and a possible hash value for fast comparison). Currently this behavior is phet-io only, so it would be good to discuss if this can be opened enough to allow this testing.

Ideally this could not only run on bayes (where it might have a "visually changed" highlight when detected), but I'd want to run this before/after local changes, so I can know if there is a bug before pushing to master.

Note that screenshots are probably very browser/platform/computer-specific. Cross-browser things would be hard to identify (as font sizes may slightly change layout, causing slight visual differences cross-platform).

[jonathanolson]

These tests would also probably identify any usage of the built-in Math.random() or things that weren't based on seeds, and would presumably ensure that phet-io playback should visually match the original.

[samreid]

Here's a node library that shows differences between images: https://github.com/uber-archive/image-diff

[pixelzoom]

8/17/17 dev meeting notes:

• add a query parameter to set random seed (see initialize-globals.js line 366)
• do fuzz test and capture screenshots
• compare screenshots using checksum, scenery code, or image-diff (above)
• do this on continuous testing server, so it's same computer and browser