Cannot run test_dir from the command line with a non-default reporter

[billdenney]

I'm having trouble tracing down exactly where the problem is coming from here, but I know some examples that reproduce the problems. The procedure to reproduce the errors (as close as I can get to a reprex) has worked on three packages that I've tested it with so far.

My overall goal is to put testing for an interacting set of packages into a script that will run locally nightly. I need to run it locally (or more exactly, not on typical CI) due to the duration and computational intensity of the tests. I then will upload the test results to track progress or new failures. (If what I'm doing is not the best process to achieve this goal, I'm also all ears.)

What worked was using the default reporter with devtools::test():

R -e "devtools::load_all();devtools::test()"

But, this has variable output based on the timing of the tests and other parts. And, I want the diff between tests to show the changes clearly. So, I wanted to have it print out every test, print out every skip, not print the timing of the results, and to run every test. For that, I tried the following:

R -e "devtools::load_all();devtools::test(reporter=testthat::ProgressReporter$new( show_praise=FALSE, min_time=Inf, update_interval=0, verbose_skips=TRUE, max_failures=Inf))"

But, that gave this error

ℹ Testing PKNCA
Error in test_dir(test_path, package = package, reporter = reporter, ...,  :
  attempt to apply non-function
Calls: <Anonymous> -> <Anonymous> -> test_dir
Execution halted

And, loading an interactive R session and running the code directly works.

That appears to be an issue here for reasons that are not clear to me:

testthat/R/test-package.R

Lines 63 to 69 in 22a68dd

	test_dir(
	test_path,
	package = package,
	reporter = reporter,
	...,
	load_package = if (package != "testthat") "source" else "none"
	)

Next, I tried to call test_dir() directly:

R -e "devtools::load_all();testthat::test_dir(path='tests/testthat', reporter=testthat::ProgressReporter$new( show_praise=FALSE, min_time=Inf, update_interval=0, verbose_skips=TRUE, max_failures=Inf))"

And, that gave the same error.

But again, trying without a reporter worked.

R -e "devtools::load_all();testthat::test_dir(path='tests/testthat')"