Skip to content

Commit

Permalink
warn about the delta method for non-linear expressions
Browse files Browse the repository at this point in the history
Fixes #821.
  • Loading branch information
huftis committed Aug 9, 2023
1 parent a198987 commit 59ac6af
Show file tree
Hide file tree
Showing 4 changed files with 19 additions and 1 deletion.
2 changes: 2 additions & 0 deletions R/hypotheses.R
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@
#'
#' Warning #2: For hypothesis tests on objects produced by the `marginaleffects` package, it is safer to use the `hypothesis` argument of the original function. Using `hypotheses()` may not work in certain environments, in lists, or when working programmatically with *apply style functions.
#'
#' Warning #3: The tests assume that the `hypothesis` expression is (approximately) normally distributed, which for non-linear functions of the parameters may not be realistic. More reliable confidence intervals can be obtained using the \code{inferences()} function with `method = "boot"`.
#'
#' @inheritParams comparisons
#' @param model Model object or object generated by the `comparisons()`, `slopes()`, `predictions()`, or `marginal_means()` functions.
#' @param FUN `NULL` or function.
Expand Down
14 changes: 14 additions & 0 deletions book/articles/hypothesis.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,20 @@ slopes(
hypothesis = "exp(am) - 2 * vs = -400")
```

But note that the p values and confidence intervals are calculated using the delta method and are thus based on the assumption that the `hypotheses` expression is approximately normally distributed. For (very) non-linear functions of the parameters, this is not realistic, and we get p values with incorrect error rates and confidence intervals with incorrect coverage probabilities. For such hypotheses, it’s better to calculate the confidence intervals using the bootstrap (see [`inferences`](reference/inferences.html) for details):

```{r}
set.seed(1234)
slopes(
mod,
newdata = "mean",
hypothesis = "exp(am) - 2 * vs = -400") |>
inferences(method = "boot")
```

While the confidence interval from the delta method is symmetric (equal to the estimate ± 1.96 times the standard error), the more reliable confidence interval from the bootstrap is (here) highly skewed.


### Adjusted Predictions

Now consider the case of adjusted predictions:
Expand Down
2 changes: 1 addition & 1 deletion book/articles/uncertainty.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ Scroll down this page to the Numerical Derivatives section to see a detailed exp

## Standard errors and intervals for `slopes()` and `comparisons()`

All standard errors for the `slopes()` and `comparisons()` functions are computed using the delta method, as described above.
All standard errors for the `slopes()` and `comparisons()` functions are computed using the delta method, as described above. The confidence intervals are calculated as estimate ± `qnorm((1 - conf_level) / 2)` standard errors (e.g., for 95% confidence intervals, estimate ± 1.96 standard errors) and assume that the (transformed) estimates are normally distributed.

## Standard errors and intervals for `marginal_means()` and `predictions()`

Expand Down
2 changes: 2 additions & 0 deletions man/hypotheses.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit 59ac6af

Please sign in to comment.