Add benchmarks to CI

[SemyonSinchenko]

It would be very cool to have some performance benchmarks in CI.
Benchmarks, mentioned in README is a nice thing to start from:
https://github.com/mrpowers-io/spark-fast-tests?tab=readme-ov-file#why-is-this-library-fast

[zeotuan]

I did some test benchmarking on this https://github.com/mrpowers-io/spark-fast-tests/pull/122/files#diff-7e79c32936db58af66efbf87c7e0114672f89209aa69722e98abee90faee72e3.
what do you think about using https://github.com/sbt/sbt-jmh

[SemyonSinchenko]

Looks cool, thank you! But we definitely should have it as a subproject.

[zeotuan]

@SemyonSinchenko I added a Draft with an initial project structure for Benchmarks module as well as an additional benchmark step for CI.
Those we still need additional work for:

• Test Data generation + Parameterized Benchmark
• Investigate most appropriate, important test metric for the project

[zeotuan]

[info] ColumnComparerBenchmark.assertColumnEqualityBenchmarks                      avgt   20    5361225.521 ±    20905.951  ns/op
[info] DataFrameComparerBenchmark.assertApproximateDataFrameEqualityWithPrecision  avgt   20   64024117.133 ±   768249.992  ns/op
[info] ColumnComparerBenchmark.assertColumnEqualityBenchmarks                        ss   20   32886862.500 ±  3606312.983  ns/op
[info] DataFrameComparerBenchmark.assertApproximateDataFrameEqualityWithPrecision    ss   20  155991154.500 ± 17649852.089  ns/op

surprisingly, our comparer is performing better than what is being shown in README:

• assertColumnEquality take 5ms/32ms (warmed up vs single shot) each run for 200 rows on simple data type
• assertApproximateDataFrameEquality take 64ms/176ms (warmed up vs single shot) each run for 100 rows and 3 columns (though usual test will probably run on DFs that are much wider)

I will temporarily disable average benchmark mode until we can come up with a better plan to run.
Probably we will do a hybrid approach in the future where single shot which is only a few seconds is done on push and the whole suite will run on merge or maybe release flow as @alfonsorr suggested

[SemyonSinchenko]

@zeotuan Can we anyhow save results? For example, can we create another branch and do force-push from CI with results in form of MD-files or even plain text?

[zeotuan]

I can make jmh store result in json (or csv and some other format) and have github action upload the artifact which we can download in same or different workflow for updating the MD file.
Should we use github action for auto commit. Maybe auto raise PR for updating MD is a safer option

[SemyonSinchenko]

I can make jmh store result in json (or csv and some other format) and have github action upload the artifact which we can download in same or different workflow for updating the MD file.
Should we use github action for auto commit. Maybe auto raise PR for updating MD is a safer option

My idea was to make a simple script in scala that parse the result and use them with a templating tool to fill the markdown file. Then, the file will be a part of the existing static website.

[zeotuan]

Sure, I think either way it's a two step process:

• have a separate Workflow that do all the benchmark and publish the result artifact (this can be access by same or different workflow and even Rest API)
• download all the result and run our scala tool to parse and aggregate the result and produce them for the templating tool

[SemyonSinchenko]

Sure, I think either way it's a two step process:

• have a separate Workflow that do all the benchmark and publish the result artifact (this can be access by same or different workflow and even Rest API)
• download all the result and run our scala tool to parse and aggregate the result and produce them for the templating tool

I just thought that maybe we can do it even easier: we can run such a script in the docs ci, fill the template, and for the docs ci it will be just like an md file...

[zeotuan]

I see, so no need to storing intermediate result. I have no objection