rbmi 1.3.1: first fit after compiling model not reproducible with subsequent fits

[luwidmer]

Describe the bug
When fitting the rbmi model for the first time and the model is initially compiled, the results differ from subsequent fits. Is it possible the compilation process has some side effects (e.g., changing the seed)?

Update: it does indeed look like the seed that is set is different.

To Reproduce

library(rbmi)
options("rbmi.cache_dir" = tempdir(check = TRUE))
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union


data("antidepressant_data")
dat <- antidepressant_data
dat <- expand_locf(
  dat,
  PATIENT = levels(dat$PATIENT), # expand by PATIENT and VISIT
  VISIT = levels(dat$VISIT),
  vars = c("BASVAL", "THERAPY"), # fill with LOCF BASVAL and THERAPY
  group = c("PATIENT"),
  order = c("PATIENT", "VISIT")
)
dat_ice <- dat |>
  arrange(PATIENT, VISIT) |>
  filter(is.na(CHANGE)) |>
  group_by(PATIENT) |>
  slice(1) |>
  ungroup() |>
  select(PATIENT, VISIT) |>
  mutate(strategy = "JR")
dat_ice <- dat_ice[-which(dat_ice$PATIENT == 3618), ]
vars <- set_vars(
  outcome = "CHANGE",
  visit = "VISIT",
  subjid = "PATIENT",
  group = "THERAPY",
  covariates = c("BASVAL*VISIT", "THERAPY*VISIT")
)
method <- method_bayes(
  burn_in = 100,
  burn_between = 1,
  n_samples = 200
)

sample_and_compute_poolobj <- function(dat, dat_ice, vars, method){
  trace(what = rbmi:::fit_mcmc, at = 12, tracer = quote(print(sampling_args$seed)))
  drawObj <- draws(
    data = dat,
    data_ice = dat_ice,
    vars = vars,
    method = method,
    quiet = TRUE
  )
  
  # Impute the data
  imputeObj <- impute(
    drawObj,
    references = c("DRUG" = "PLACEBO", "PLACEBO" = "PLACEBO")
  )
  
  # Fit the analysis model on each imputed dataset
  anaObj <- analyse(
    imputeObj,
    ancova,
    vars = set_vars(
      subjid = "PATIENT",
      outcome = "CHANGE",
      visit = "VISIT",
      group = "THERAPY",
      covariates = c("BASVAL")
    )
  )
  
  # Apply a delta adjustment
  
  # Add a delta-value of 5 to all imputed values (i.e. those values
  # which were missing in the original dataset) in the drug arm.
  delta_df <- delta_template(imputeObj) %>%
    as_tibble() %>%
    mutate(delta = if_else(THERAPY == "DRUG" & is_missing , 5, 0)) %>%
    select(PATIENT, VISIT, delta)
  
  # Repeat the analyses with the adjusted values
  anaObj_delta <- analyse(
    imputeObj,
    ancova,
    delta = delta_df,
    vars = set_vars(
      subjid = "PATIENT",
      outcome = "CHANGE",
      visit = "VISIT",
      group = "THERAPY",
      covariates = c("BASVAL")
    )
  )
  
  # Pool the results
  poolObj <- pool(
    anaObj,
    conf.level = 0.95,
    alternative = "two.sided"
  )
  poolObj
}
set.seed(123)
poolObj1 <- sample_and_compute_poolobj(dat, dat_ice, vars, method)
#> Tracing function "fit_mcmc" in package "rbmi (not-exported)"
#> Tracing fit_mcmc(designmat = model_df_scaled[, -1, drop = FALSE], outcome = model_df_scaled[,  .... step 12 
#> [1] 1673048412
#> Warning: The Effective Sample Size is below 40% for 8 parameters. Please
#> consider increasing burn-in and/or burn-between, or the number of samples
set.seed(123)
poolObj2 <- sample_and_compute_poolobj(dat, dat_ice, vars, method)
#> Tracing function "fit_mcmc" in package "rbmi (not-exported)"
#> recompiling to avoid crashing R session
#> Tracing fit_mcmc(designmat = model_df_scaled[, -1, drop = FALSE], outcome = model_df_scaled[,  .... step 12 
#> [1] 1235143119
#> Warning: The Effective Sample Size is below 40% for 15 parameters. Please
#> consider increasing burn-in and/or burn-between, or the number of samples
set.seed(123)
poolObj3 <- sample_and_compute_poolobj(dat, dat_ice, vars, method)
#> Tracing function "fit_mcmc" in package "rbmi (not-exported)"
#> recompiling to avoid crashing R session
#> Tracing fit_mcmc(designmat = model_df_scaled[, -1, drop = FALSE], outcome = model_df_scaled[,  .... step 12 
#> [1] 1235143119
#> Warning: The Effective Sample Size is below 40% for 15 parameters. Please
#> consider increasing burn-in and/or burn-between, or the number of samples

print(list(
  cache_dir = getOption("rbmi.cache_dir"),
  poolObj1 = digest::digest(poolObj1),
  poolObj2 = digest::digest(poolObj2),
  poolObj3 = digest::digest(poolObj3)
))
#> $cache_dir
#> [1] "/var/folders/cd/0t58mmhj4nzgsn343pkx2lc80000gn/T//Rtmp8DfeuY"
#> 
#> $poolObj1
#> [1] "fc1a471c915b5a421c845c1bd201ec48"
#> 
#> $poolObj2
#> [1] "7570c9b6c00bc45cdde9b4b07ca82d67"
#> 
#> $poolObj3
#> [1] "7570c9b6c00bc45cdde9b4b07ca82d67"

^{Created on 2025-01-06 with reprex v2.1.1}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.4.1 (2024-06-14)
#>  os       macOS 15.2
#>  system   aarch64, darwin20
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       Europe/Zurich
#>  date     2025-01-06
#>  pandoc   3.2 @ /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package      * version  date (UTC) lib source
#>  assertthat     0.2.1    2019-03-21 [1] CRAN (R 4.4.0)
#>  backports      1.5.0    2024-05-23 [1] CRAN (R 4.4.0)
#>  callr          3.7.6    2024-03-25 [1] CRAN (R 4.4.0)
#>  checkmate      2.3.2    2024-07-29 [1] CRAN (R 4.4.0)
#>  cli            3.6.3    2024-06-21 [1] CRAN (R 4.4.0)
#>  codetools      0.2-20   2024-03-31 [1] CRAN (R 4.4.1)
#>  colorspace     2.1-1    2024-07-26 [1] CRAN (R 4.4.0)
#>  curl           6.0.1    2024-11-14 [1] CRAN (R 4.4.1)
#>  digest         0.6.37   2024-08-19 [1] CRAN (R 4.4.1)
#>  dplyr        * 1.1.4    2023-11-17 [1] CRAN (R 4.4.0)
#>  evaluate       1.0.1    2024-10-10 [1] CRAN (R 4.4.1)
#>  fansi          1.0.6    2023-12-08 [1] CRAN (R 4.4.0)
#>  fastmap        1.2.0    2024-05-15 [1] CRAN (R 4.4.0)
#>  fs             1.6.5    2024-10-30 [1] CRAN (R 4.4.1)
#>  generics       0.1.3    2022-07-05 [1] CRAN (R 4.4.0)
#>  ggplot2        3.5.1    2024-04-23 [1] CRAN (R 4.4.0)
#>  glue           1.8.0    2024-09-30 [1] CRAN (R 4.4.1)
#>  gridExtra      2.3      2017-09-09 [1] CRAN (R 4.4.0)
#>  gtable         0.3.6    2024-10-25 [1] CRAN (R 4.4.1)
#>  htmltools      0.5.8.1  2024-04-04 [1] CRAN (R 4.4.0)
#>  inline         0.3.20   2024-11-10 [1] CRAN (R 4.4.1)
#>  jsonlite       1.8.9    2024-09-20 [1] CRAN (R 4.4.1)
#>  knitr          1.49     2024-11-08 [1] CRAN (R 4.4.1)
#>  lattice        0.22-6   2024-03-20 [1] CRAN (R 4.4.1)
#>  lifecycle      1.0.4    2023-11-07 [1] CRAN (R 4.4.0)
#>  loo            2.8.0    2024-07-03 [1] CRAN (R 4.4.0)
#>  magrittr       2.0.3    2022-03-30 [1] CRAN (R 4.4.0)
#>  Matrix         1.7-1    2024-10-18 [1] CRAN (R 4.4.1)
#>  matrixStats    1.4.1    2024-09-08 [1] CRAN (R 4.4.1)
#>  mmrm           0.3.14   2024-09-27 [1] CRAN (R 4.4.1)
#>  munsell        0.5.1    2024-04-01 [1] CRAN (R 4.4.0)
#>  nlme           3.1-166  2024-08-14 [1] CRAN (R 4.4.0)
#>  pillar         1.9.0    2023-03-22 [1] CRAN (R 4.4.0)
#>  pkgbuild       1.4.5    2024-10-28 [1] CRAN (R 4.4.1)
#>  pkgconfig      2.0.3    2019-09-22 [1] CRAN (R 4.4.0)
#>  processx       3.8.4    2024-03-16 [1] CRAN (R 4.4.0)
#>  ps             1.8.1    2024-10-28 [1] CRAN (R 4.4.1)
#>  QuickJSR       1.4.0    2024-10-01 [1] CRAN (R 4.4.1)
#>  R6             2.5.1    2021-08-19 [1] CRAN (R 4.4.0)
#>  rbibutils      2.3      2024-10-04 [1] CRAN (R 4.4.1)
#>  rbmi         * 1.3.1    2024-12-11 [1] CRAN (R 4.4.1)
#>  Rcpp           1.0.13-1 2024-11-02 [1] CRAN (R 4.4.1)
#>  RcppParallel   5.1.9    2024-08-19 [1] CRAN (R 4.4.1)
#>  Rdpack         2.6.2    2024-11-15 [1] CRAN (R 4.4.1)
#>  reprex         2.1.1    2024-07-06 [1] CRAN (R 4.4.0)
#>  rlang          1.1.4    2024-06-04 [1] CRAN (R 4.4.0)
#>  rmarkdown      2.29     2024-11-04 [1] CRAN (R 4.4.1)
#>  rstan          2.32.6   2024-03-05 [1] CRAN (R 4.4.0)
#>  rstudioapi     0.17.1   2024-10-22 [1] CRAN (R 4.4.1)
#>  scales         1.3.0    2023-11-28 [1] CRAN (R 4.4.0)
#>  sessioninfo    1.2.2    2021-12-06 [1] CRAN (R 4.4.0)
#>  StanHeaders    2.32.10  2024-07-15 [1] CRAN (R 4.4.0)
#>  stringi        1.8.4    2024-05-06 [1] CRAN (R 4.4.0)
#>  stringr        1.5.1    2023-11-14 [1] CRAN (R 4.4.0)
#>  tibble         3.2.1    2023-03-20 [1] CRAN (R 4.4.0)
#>  tidyselect     1.2.1    2024-03-11 [1] CRAN (R 4.4.0)
#>  TMB            1.9.15   2024-09-09 [1] CRAN (R 4.4.1)
#>  utf8           1.2.4    2023-10-22 [1] CRAN (R 4.4.0)
#>  V8             6.0.0    2024-10-12 [1] CRAN (R 4.4.1)
#>  vctrs          0.6.5    2023-12-01 [1] CRAN (R 4.4.0)
#>  withr          3.0.2    2024-10-28 [1] CRAN (R 4.4.1)
#>  xfun           0.49     2024-10-31 [1] CRAN (R 4.4.1)
#>  yaml           2.3.10   2024-07-26 [1] CRAN (R 4.4.0)
#> 
#>  [1] /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: Multiple
• R version: 4.3.1, 4.4.1
• rbmi version: 1.3.1

[luwidmer]

Interestingly, if one saves the random seed in get_stan_model() and restores it at the end of the function, things become deterministic, i.e., adding backup_seed <- .Random.seed at the beginning and modify the end to be

    result <- rstan::stan_model(
        file = model_file,
        auto_write = TRUE,
        model_name = "rbmi_mmrm"
    )

    .Random.seed <<- backup_seed

    return(result)

seems to fix it. Not sure what in get_stan_model() non-deterministically modifies the seed though.

[luwidmer]

Apparently stan_model has a side effect the first time it compiles the model - printing digest::digest(.Random.seed) illustrates this:

[1] "Before stan_model: b700221453276a0bd3834d5db175658c"
[1] "After stan_model: 58fee3d740d718493cdffb8cf12add63"
Tracing fit_mcmc(designmat = model_df_scaled[, -1, drop = FALSE], outcome = model_df_scaled[,  .... step 12 
[1] "2858edd9e7d6dd4b196ecb1e854d449f"
Tracing function "fit_mcmc" in package "rbmi (not-exported)"
Tracing function "fit_mcmc" in package "rbmi (not-exported)"
[1] "b700221453276a0bd3834d5db175658c"
[1] "b700221453276a0bd3834d5db175658c"
[1] "b700221453276a0bd3834d5db175658c"
[1] "b700221453276a0bd3834d5db175658c"
[1] "Before stan_model: b700221453276a0bd3834d5db175658c"
recompiling to avoid crashing R session
[1] "After stan_model: b700221453276a0bd3834d5db175658c"

[luwidmer]

Apparently this comes from processx::get_id(), see

Browse[1]> n
debug: private$tree_id <- get_id()
Browse[1]> print(digest::digest(.Random.seed))
[1] "b700221453276a0bd3834d5db175658c"
Browse[1]> n
debug: if (!is.null(wd)) {
    wd <- normalizePath(wd, winslash = "\\", mustWork = FALSE)
}
Browse[1]> print(digest::digest(.Random.seed))
[1] "58fee3d740d718493cdffb8cf12add63"

The call stack for this one is quite deep:

Browse[1]> lobstr::cst()
     ▆
  1. └─global sample_and_compute_poolobj(dat, dat_ice, vars, method) at ~/Downloads/test_rbmi_131.R:114:3
  2.   ├─rbmi::draws(...) at ~/Downloads/test_rbmi_131.R:55:5
  3.   └─rbmi:::draws.bayes(...)
  4.     └─rbmi:::fit_mcmc(...)
  5.       └─rbmi:::get_stan_model()
  6.         └─rstan::stan_model(...)
  7.           └─rstan:::cxxfunctionplus(...)
  8.             └─pkgbuild::has_build_tools(debug = FALSE)
  9.               └─pkgbuild::has_compiler(debug = debug)
 10.                 ├─base::tryCatch(...)
 11.                 │ └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 12.                 │   └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 13.                 │     └─base (local) doTryCatch(return(expr), name, parentenv, handler)
 14.                 └─callr::rcmd_safe(...)
 15.                   └─callr:::run_r(options)
 16.                     ├─base::with(...)
 17.                     └─base::with.default(...)
 18.                       └─base::eval(substitute(expr), data, enclos = parent.frame())
 19.                         └─base::eval(substitute(expr), data, enclos = parent.frame())
 20.                           ├─callr:::with_envvar(...)
 21.                           │ └─base::force(code)
 22.                           ├─base::do.call(...)
 23.                           └─processx (local) `<fn>`(...)
 24.                             └─process$new(...)
 25.                               └─processx (local) initialize(...)
 26.                                 └─processx:::process_initialize(...)
 27.                                   └─lobstr::cst()

[luwidmer]

Apparently this is a known issue in processx: r-lib/processx#386 (and was discovered in pkgbuild, too: r-lib/pkgbuild#185)

[luwidmer]

Can confirm that installing the latest processx off Github (remotes::install_github("r-lib/processx")) fixes this

[gowerc]

@luwidmer - Thanks for the detailed bug report + breakdown !

My gut feeling is that even if this is fixed in the upstream that we should probably apply some fix within rbmi as well as many R setups will have frozen R packages meaning they will still be exposed to this issue.

I'm guessing we could just move the random int sampling to be within the method_bayes() function and then propagate the value down, that way it occurs before any of the other code gets a chance to run.

[luwidmer]

I tired allocating the seed before, however, this still did not guarantee that the results were the same (at least not in the bitwise sense) - saving .Random.seed and restoring it around the compilation process did work, though

[gowerc]

@luwidmer - Interesting ...

I'm semi-guessing but I would assume that comes from the imputation stage when we draw imputed values from the conditional normal. I'd be curious if you re-set the seed before impute() in your example if that then guarantees reproducibility (I can test myself later if you don't have time).

I guess even if that works its not ideal, realistically users should just set the seed once before the first draws() call and then not have to think about it. Yer I guess saving/restoring .Random.seed would be our only real option then. I'd probably just do it around the Stan compilation code though as we know that is the problem area and I'd be worried about unforeseen consequences from potential future updates if we widened the area too much.

[luwidmer]

Agree. I was also a bit surprised that this has been fixed on processx Github since half a year and is still broken on CRAN

[luwidmer]

https://cran.r-project.org/web/packages/processx/index.html processx 3.8.5 fixes this too now :)