Filter out unreproducible performance spikes

[pitag-ha]

When running ocamlmerlin, we unfrequently yet consistently experience unreproducible performance outliers, meaning the following: When running the exact same ocamlmerlin command multiple times, one time it takes considerably longer than all the other times. To address this, we can implement a merl-an option to filter those outliers out.

Details on the task

merl-an already supports a workflow to run the same ocamlmerlin query on the same sample multiple times. However, for the benchmark command, each query on a sample is currently run only one time (see Repeats 1). So the first step will be to set that number to 5 or 10.

Once that's done, we can introduce a new cli-flag --filter-outliers to the merl-an benchmark and the merl-an performance command. Activating that flag, should filter out potential outliers from the list of responses for one sample (this list contains the responses of running the same ocamlmerlin query on the same sample multiple times). The filtering on that list can be done according to the following case distinction:

• If init_cache is true, we'll have a look at the whole list of reponses. For every item in the list, we have a look at the timing component. If the longest timing is five times greater than the second longest, then we filter it out.
• If init_cache is false, then we do the same thing, but instead of having a look at the whole list, we ignore the first item in the list (explanation: for that item, it will be expected that it takes longer than the rest, since the cache potentially wasn't warm yet).

Concrete motivation for the task

We're using merl-an benchmark for Merlin's current-bench performance CI set-up. current-bench visualizes the data it receives in graphs, scaled up to the maximum value it receives. So unreproducible performance outliers distort the scaling to an extent that the visualization becomes useless. To fix this, we need to filter out the outliers.