Expectation errors should be based on median run values

[tshen8]

When performing multiple runs on a page with expectation set, the entire testing process will fail if any one of the runs exceeds any expectation value.

If the purpose of doing multiple runs is to gather several data points for performance metrics in order to rule out outliers and unrepresentative numbers, I think it makes sense that the pass/fail should be based on the median numbers for the runs (which are displayed at the end anyways).

What do you guys think? I'm happy to make a PR if we're okay with this.

[denar90]

Hi, thank you for raising this question, I think it's a good idea 👍
Feel free to send PR ;)

[denar90]

should be fixed via #178

[brophdawg11]

Any idea when #178 might be merged?

[denar90]

Hi, @brophdawg11. Hope really soon, but need some polishing. Working on that...

[denar90]

closing since #178 merged