Check if parallel testing reduces check time in CI

[IndrajeetPatil]

Related to #2295

[IndrajeetPatil]

On main for macOS and R-release:

* checking tests ...
  Running ‘testthat.R’ [67s/69s]
 [68s/69s] OK

In this PR for the same config:

* checking tests ...
  Running ‘testthat.R’ [85s/46s]
 [85s/46s] OK

I don't understand why there are two timings, and which one we should care about.

The test-package workflows gives the exact duration, but that workflow fails here because options(warn = 2) seems to mess with the parallel testing.

[MichaelChirico]

Part of the issue is that I think there's a maximum number of jobs that can be running at once -- speeding up individual jobs helps some, but I'd really like to reduce the number of jobs if some jobs are redundant.

[IndrajeetPatil]

Agreed. But what do you think about turning on the parallel testing for {lintr}, in general?
Do you see any downsides to it, apart from the failing workflow, which is likely to be a {testthat} bug?

[MichaelChirico]

if we could have it run in serial by default and parallel according to environment variable on GHA only (not CRAN) I think that's preferable. parallelism is often a trojan horse for headaches if we don't get a huge benefit

[IndrajeetPatil]

Will setting something like this in testthat.R work (parallel on for non-CRAN, but off for CRAN)?

if (Sys.getenv("NOT_CRAN", "true") == "true") {
  Sys.setenv("TESTTHAT_PARALLEL" = "FALSE")
}

[IndrajeetPatil]

Filed an issue upstream about the error we see in the vigilant workflow: r-lib/testthat#1912

[AshesITR]

On main for macOS and R-release:

* checking tests ...

  Running ‘testthat.R’ [67s/69s]

 [68s/69s] OK

In this PR for the same config:

* checking tests ...

  Running ‘testthat.R’ [85s/46s]

 [85s/46s] OK

I don't understand why there are two timings, and which one we should care about.

I suspect the first is CPU time and the second is elapsed (wall) time.
So we pay 6s more computational cost for 23s time savings. Seems worth it.

[IndrajeetPatil]

The reduction in test time from parallel testing has reduced significantly since the last time we checked; probably because our test suite itself has somehow got faster 👀

• for main

* checking tests ...
  Running ‘testthat.R’ [28s/29s]
 [29s/29s] OK

• for PR

* checking tests ...
  Running ‘testthat.R’ [49s/24s]
 [49s/24s] OK

I am no longer sure if it is still worth it to turn on parallel testing, or if we'd wait for our test suite to get even bigger before we could reap the benefits of parallel testing.

[IndrajeetPatil]

@MichaelChirico, @AshesITR WDYT?

Should we keep this PR open for a few more months or a year, rebase once in a while, and check for time saving benefits?
Or should we just kill it now and revisit only when we are faced with slow tests because of the growth in test suite?

[AshesITR]

I'm fine with either way. It looks like most of the work is done already and rebasing from time to time isn't difficult.

What do you think is best?

[IndrajeetPatil]

What do you think is best?

I think it should be fine (and costless) to keep this pending for a while.