Is check-test-warnings broken?

[etiennebacher]

Following easystats/datawizard#465, I removed all tests but one in datawizard and put a warning() call inside:

test_that("standardize.lm", {
  warning("foo")
  expect_equal(1+1, 2)
})

Then I ran:

options(crayon.enabled = TRUE, warn = 2L)
devtools::test()

and it showed a test warning but not a failure:

ℹ Testing datawizard
Starting 1 test process
✔ | F W  S  OK | Context
✔ |   1      1 | standardize_models                                                                                     
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Warning (test-standardize_models.R:3:3): standardize.lm
foo
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                                        
══ Results ═════════════════════════════════════════════════════════════════════════════════════════════════════════════
Duration: 1.6 s

[ FAIL 0 | WARN 1 | SKIP 0 | PASS 1 ]

Replacing warning() by stop() correctly shows a test failure.

[IndrajeetPatil]

I can confirm that this workflow is broken. The way to make this work is to use testthat::test_dir() with options(warn = 2L). But this causes problems if parallel testing is turned on, which I suspect is a bug in {testthat}.

[IndrajeetPatil]

Bug report filed upstream r-lib/testthat#1912

[IndrajeetPatil]

Closed in daf6ede