diff --git a/R/hypotheses.R b/R/hypotheses.R index 28810f628..308355db7 100644 --- a/R/hypotheses.R +++ b/R/hypotheses.R @@ -13,6 +13,8 @@ #' #' Warning #2: For hypothesis tests on objects produced by the `marginaleffects` package, it is safer to use the `hypothesis` argument of the original function. Using `hypotheses()` may not work in certain environments, in lists, or when working programmatically with *apply style functions. #' +#' Warning #3: The tests assume that the `hypothesis` expression is (approximately) normally distributed, which for non-linear functions of the parameters may not be realistic. More reliable confidence intervals can be obtained using the \code{inferences()} function with `method = "boot"`. +#' #' @inheritParams comparisons #' @param model Model object or object generated by the `comparisons()`, `slopes()`, `predictions()`, or `marginal_means()` functions. #' @param FUN `NULL` or function. diff --git a/book/articles/hypothesis.qmd b/book/articles/hypothesis.qmd index 20dcdb658..0806d3a10 100644 --- a/book/articles/hypothesis.qmd +++ b/book/articles/hypothesis.qmd @@ -226,6 +226,20 @@ slopes( hypothesis = "exp(am) - 2 * vs = -400") ``` +But note that the p values and confidence intervals are calculated using the delta method and are thus based on the assumption that the `hypotheses` expression is approximately normally distributed. For (very) non-linear functions of the parameters, this is not realistic, and we get p values with incorrect error rates and confidence intervals with incorrect coverage probabilities. For such hypotheses, it’s better to calculate the confidence intervals using the bootstrap (see [`inferences`](reference/inferences.html) for details): + +```{r} +set.seed(1234) +slopes( + mod, + newdata = "mean", + hypothesis = "exp(am) - 2 * vs = -400") |> + inferences(method = "boot") +``` + +While the confidence interval from the delta method is symmetric (equal to the estimate ± 1.96 times the standard error), the more reliable confidence interval from the bootstrap is (here) highly skewed. + + ### Adjusted Predictions Now consider the case of adjusted predictions: diff --git a/book/articles/uncertainty.qmd b/book/articles/uncertainty.qmd index 40bc730b0..d557a14c3 100644 --- a/book/articles/uncertainty.qmd +++ b/book/articles/uncertainty.qmd @@ -38,7 +38,7 @@ Scroll down this page to the Numerical Derivatives section to see a detailed exp ## Standard errors and intervals for `slopes()` and `comparisons()` -All standard errors for the `slopes()` and `comparisons()` functions are computed using the delta method, as described above. +All standard errors for the `slopes()` and `comparisons()` functions are computed using the delta method, as described above. The confidence intervals are calculated as estimate ± `qnorm((1 - conf_level) / 2)` standard errors (e.g., for 95% confidence intervals, estimate ± 1.96 standard errors) and assume that the (transformed) estimates are normally distributed. ## Standard errors and intervals for `marginal_means()` and `predictions()` diff --git a/man/hypotheses.Rd b/man/hypotheses.Rd index 728022c7b..022643742 100644 --- a/man/hypotheses.Rd +++ b/man/hypotheses.Rd @@ -115,6 +115,8 @@ package website, or scroll down this page for a full list of vignettes: Warning #1: Tests are conducted directly on the scale defined by the \code{type} argument. For some models, it can make sense to conduct hypothesis or equivalence tests on the \code{"link"} scale instead of the \code{"response"} scale which is often the default. Warning #2: For hypothesis tests on objects produced by the \code{marginaleffects} package, it is safer to use the \code{hypothesis} argument of the original function. Using \code{hypotheses()} may not work in certain environments, in lists, or when working programmatically with *apply style functions. + +Warning #3: The tests assume that the \code{hypothesis} expression is (approximately) normally distributed, which for non-linear functions of the parameters may not be realistic. More reliable confidence intervals can be obtained using the \code{inferences()} function with \code{method = "boot"}. } \section{Joint hypothesis tests}{