reports.resultDir configuration from code does not work as depicted

[lordson-ajira]

I have been trying to set up a test which writes test result reports to multiple directories, following the example mentioned in the below file

scalameter-examples/include-statements/src/bench/scala/org/scalameter/examples/collection-benchmarks.scala

Line 59 in 41e170f

reports.resultDir -> "target/benchmarks/memory"

I end up getting only the last test report written into target/benchmarks/report, not the directories specified in code

Following scalameter/scalameter#189, scalameter/scalameter#195
and finally scalameter/scalameter@0c54382#diff-89ba343b74c4e69e1de52032524e5313666081e7813e756a254cabc697ac7e15

led me to believe that either

• The docs that mention resultDir can be changed from within code were valid at one point of time and the particular commit overwrote intended functionality and nobody caught the bug
• The docs mentioning that resultDir is a global test parameter is right and the examples are outdated, and the particular PR to resolve the issue addresses only the regression data persistor but not the HtmlReport generator

My questions now would be:

• Is it possible to support resultDir configuration in code and not through command line?
  (so I can generate reports in multiple directories with a single command line invocation)
  -- P.S. I am willing to work on a PR if it's currently unsupported but there are no objections to support resultDir configuration in code

• In case we do not want to support resultDir configuration by code, should we update the examples to point out that the setup specified in the above file is no longer supported?