Prism benchmarks

[egiurleo]

Motivation

This PR introduces a script to run benchmarks comparing the performance of the Prism parser to the Sorbet parser. The results will be saved in JSON files, allowing us to compare the performance of the Prism and Sorbet parser at any given time, as well as over time.

Benchmark Results

Benchmark 1: yjit-bench, parser only

Benchmark 1: bazel-bin/main/sorbet --parser=sorbet --stop-after=parser ../yjit-bench/benchmarks
  Time (mean ± σ):     13.984 s ±  0.375 s    [User: 22.977 s, System: 0.735 s]
  Range (min … max):   13.452 s … 14.834 s    10 runs

Benchmark 2: bazel-bin/main/sorbet --parser=prism --stop-after=parser ../yjit-bench/benchmarks
  Time (mean ± σ):      8.630 s ±  0.102 s    [User: 11.003 s, System: 0.938 s]
  Range (min … max):    8.504 s …  8.856 s    10 runs

Summary
  bazel-bin/main/sorbet --parser=prism --stop-after=parser ../yjit-bench/benchmarks ran
    1.62 ± 0.05 times faster than bazel-bin/main/sorbet --parser=sorbet --stop-after=parser ../yjit-bench/benchmarks

Benchmark 2: sorbet-test, parser only

Benchmark 1: bazel-bin/main/sorbet --parser=sorbet --stop-after=parser test/testdata
  Time (mean ± σ):     10.043 s ±  0.025 s    [User: 15.975 s, System: 1.059 s]
  Range (min … max):   10.008 s … 10.082 s    10 runs

  Warning: Ignoring non-zero exit code.

Benchmark 2: bazel-bin/main/sorbet --parser=prism --stop-after=parser test/testdata
  Time (mean ± σ):      9.029 s ±  0.117 s    [User: 12.662 s, System: 1.951 s]
  Range (min … max):    8.934 s …  9.251 s    10 runs

Summary
  bazel-bin/main/sorbet --parser=prism --stop-after=parser test/testdata ran
    1.11 ± 0.01 times faster than bazel-bin/main/sorbet --parser=sorbet --stop-after=parser test/testdata

Benchmark 3: prism regression tests, whole pipeline

Benchmark 1: bazel-bin/main/sorbet --parser=sorbet test/prism_regression
  Time (mean ± σ):     841.8 ms ±   8.1 ms    [User: 1593.8 ms, System: 119.3 ms]
  Range (min … max):   831.6 ms … 855.6 ms    10 runs

Benchmark 2: bazel-bin/main/sorbet --parser=prism test/prism_regression
  Time (mean ± σ):     841.2 ms ±   7.9 ms    [User: 1618.4 ms, System: 102.3 ms]
  Range (min … max):   831.3 ms … 854.8 ms    10 runs

Summary
  bazel-bin/main/sorbet --parser=prism test/prism_regression ran
    1.00 ± 0.01 times faster than bazel-bin/main/sorbet --parser=sorbet test/prism_regression

[egiurleo]

This was Alex M's idea -- basically warm up the file system cache by reading the benchmark files a few times. I've found that it significantly improves the accuracy of the benchmarks on the first run!

[Morriar]

TIL

[KaanOzkan]

This step lowering the variance makes me think the # of times hyperfine is warming up may not be enough. It's not crucial but may be worth playing around with different warmup values.

[egiurleo]

@KaanOzkan agreed. This is significantly faster than warmup loops, so I've done this for now, but I think it's worth reexamining the approach as we go.

[Morriar]

TIL

[amomchilov]

Great!