diff --git a/DESCRIPTION b/DESCRIPTION index 7f0e960..8e458e8 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,7 +1,7 @@ Package: adaptr Title: Adaptive Trial Simulator -Version: 1.3.2 -Date: 2023-08-21 +Version: 1.4.0 +Date: 2024-05-03 Authors@R: c(person("Anders", "Granholm", email = "andersgran@gmail.com", @@ -22,8 +22,8 @@ Authors@R: Description: Package that simulates adaptive (multi-arm, multi-stage) clinical trials using adaptive stopping, adaptive arm dropping, and/or adaptive randomisation. Developed as part of the INCEPT (Intensive Care Platform - Trial) project (), which is primarily supported by a - grant from Sygeforsikringen "danmark" (). + Trial) project (), primarily supported by a grant + from Sygeforsikringen "danmark" (). License: GPL (>= 3) Imports: stats, @@ -37,7 +37,7 @@ URL: https://inceptdk.github.io/adaptr/, https://github.com/INCEPTdk/adaptr/, https://incept.dk/ BugReports: https://github.com/INCEPTdk/adaptr/issues/ -RoxygenNote: 7.2.3 +RoxygenNote: 7.3.1 Suggests: ggplot2, covr, diff --git a/NAMESPACE b/NAMESPACE index 4ab778e..6b7ce82 100644 --- a/NAMESPACE +++ b/NAMESPACE @@ -25,6 +25,7 @@ export(setup_cluster) export(setup_trial) export(setup_trial_binom) export(setup_trial_norm) +export(update_saved_calibration) export(update_saved_trials) import(parallel) importFrom(stats,aggregate) diff --git a/NEWS.md b/NEWS.md index 83fa67e..09db577 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,3 +1,64 @@ +# adaptr 1.4.0 + +This is a minor release implementing new functionality, and including bug fixes, +updates to documentation, argument checking and test coverage. + +### New features and major changes: + +* Added the `rescale_probs` argument to the `setup_trial()` family of + functions, allowing automatic rescaling of fixed allocation probabilities + and or minimum/maximum allocation probability limits when arms are dropped + in simulations of trial designs with `>2 arms`. + +* The `extract_results()` function now also returns errors for each simulation + (in addition to squared errors) and the `check_performance()`, + `plot_convergence()`, and `summary()` functions (including their `print()` + methods) now calculate and present median absolute errors (in addition to + root mean squared errors). + +* The `plot_metrics_ecdf()` function now supports plotting errors (raw, + squared, and absolute), and now takes the necessary additional arguments + passed to `extract_results()` used for arm selection in simulated trials not + stopped for superiority. + +* Added the `update_saved_calibration()` function to update calibrated trial + objects (including embedded trial specifications and results) saved by the + `calibrate_trial()` function using previous versions of the package. + +* Rewritten README and 'Overview' vignette to better reflect the typical usage + and workflow. + +### Minor changes and bug fixes: + +* The `setup_trial()` family of functions now stops with an error if less than + two `arms` are provided. + +* The `setup_trial()` family of functions now stops with an error if + `control_prob_fixed` is `"match"` and `fixed_probs` is provided for the + common control arm. + +* Improved error message when `true_ys`-argument is missing in + `setup_trial_binom()` or when `true_ys`- or `sds`-argument is missing in + `setup_trial_norm()`. + +* Changed the number of rows used in `plot_convergence()` and `plot_status()` + if the total number of plots is `<= 3` and `nrow` and `ncol` are `NULL`. + +* Fixed a bug in `extract_results()` (and thus all functions relying on it), + causing arm selection in inconclusive trial simulations to error when + stopped for practical equivalence and more simulated patients were + randomised than included in the last analysis. + +* Improved test coverage. + +* Minor edits and clarification to package documentation. + +* Added references to two open access articles (with code) of simulation + studies using `adaptr` to assess the performance of adaptive clinical trials + according to different follow-up/data collection lags + () and different sceptical priors + () + # adaptr 1.3.2 This is a patch release with bug fixes and documentation updates. diff --git a/R/adaptr-package.R b/R/adaptr-package.R index 5146d23..407ee11 100644 --- a/R/adaptr-package.R +++ b/R/adaptr-package.R @@ -1,9 +1,5 @@ #' adaptr: Adaptive Trial Simulator #' -#' @docType package -#' @name adaptr-package -#' @aliases adaptr -#' #' @description #' \if{html}{ #' \figure{adaptr.png}{options: width="120" alt="logo"} @@ -76,6 +72,16 @@ #' #' [GitHub repository](https://github.com/INCEPTdk/adaptr/) #' +#' **Examples of studies using `adaptr`:** +#' +#' Granholm A, Lange T, Harhay MO, Jensen AKG, Perner A, Møller MH, Kaas-Hansen +#' BS (2023). Effects of duration of follow-up and lag in data collection on the +#' performance of adaptive clinical trials. Pharm Stat. \doi{10.1002/pst.2342} +#' +#' Granholm A, Lange T, Harhay MO, Perner A, Møller MH, Kaas-Hansen BS (2024). +#' Effects of sceptical priors on the performance of adaptive clinical trials +#' with binary outcomes. Pharm Stat. \doi{10.1002/pst.2387} +#' #' #' @seealso #' [setup_cluster()], [setup_trial()], [setup_trial_binom()], @@ -84,4 +90,4 @@ #' [check_remaining_arms()], [plot_convergence()], [plot_metrics_ecdf()], #' [print()], [plot_status()], [plot_history()]. #' -NULL +"_PACKAGE" diff --git a/R/calibrate_trial.R b/R/calibrate_trial.R index e56b883..6bd8dbb 100644 --- a/R/calibrate_trial.R +++ b/R/calibrate_trial.R @@ -555,11 +555,16 @@ calibrate_trial <- function( # Check if a previous version should be loaded and returned (only if overwrite is FALSE) if (ifelse(!is.null(path) & !overwrite, file.exists(path), FALSE)) { prev <- readRDS(path) - # Compare previous/current trial_specs + # Compare previous/current objects prev_spec_nofun <- prev$input_trial_spec spec_nofun <- trial_spec prev_spec_nofun$fun_y_gen <- prev_spec_nofun$fun_draws <- prev_spec_nofun$fun_raw_est <- spec_nofun$fun_y_gen <- spec_nofun$fun_draws <- spec_nofun$fun_raw_est <- NULL - if (!isTRUE(all.equal(prev_spec_nofun, spec_nofun)) | + # Compare + if ((prev$adaptr_version != .adaptr_version)) { # Check version + stop0("The object in path was created by a previous version of adaptr and ", + "cannot be used by this version of adaptr unless the object is updated. ", + "Type 'help(\"update_saved_calibration\")' for help on updating.") + } else if (!isTRUE(all.equal(prev_spec_nofun, spec_nofun)) | # Check spec besides version !equivalent_funs(prev$input_trial_spec$fun_y_gen, trial_spec$fun_y_gen) | !equivalent_funs(prev$input_trial_spec$fun_draws, trial_spec$fun_draws) | !equivalent_funs(prev$input_trial_spec$fun_raw_est, trial_spec$fun_raw_est)) { @@ -577,10 +582,6 @@ calibrate_trial <- function( } else if (!equivalent_funs(fun, prev$fun)){ stop0("The calibration function (argument 'fun') in the object in path ", "is different from the current calibration function.") - } else if ((prev$adaptr_version != .adaptr_version)) { # Check version - # Included for future use - later, stopping will only be needed when - # differences in behaviour of the used functions are required - stop0("the object in path was created by a previous version of adaptr.") } if (verbose) { message(paste0( diff --git a/R/check_performance.R b/R/check_performance.R index 6435f35..af155f6 100644 --- a/R/check_performance.R +++ b/R/check_performance.R @@ -63,9 +63,9 @@ calculate_idp <- function(sels, arms, true_ys, highest_is_best) { #' SDs), `"err_mad"` (bootstrapped MAD-SDs, as described in [setup_trial()] #' and [stats::mad()]), `"lo_ci"`, and `"hi_ci"`, the latter two corresponding #' to the lower/upper limits of the percentile-based bootstrapped confidence -#' intervals. Bootstrap estimates are **not** calculated for the mininum +#' intervals. Bootstrap estimates are **not** calculated for the minimum #' (`_p0`) and maximum values (`_p100`) of `size`, `sum_ys`, and `ratio_ys`, -#' as non-parametric bootstrapping for mininum/maximum values is not +#' as non-parametric bootstrapping for minimum/maximum values is not #' sensible - bootstrap estimates for these values will be `NA`.\cr #' The following performance metrics are calculated: #' \itemize{ @@ -104,7 +104,10 @@ calculate_idp <- function(sels, arms, true_ys, highest_is_best) { #' selection, according to the specified selection strategy. Contains one #' element per `arm`, named `prob_select_arm_` and #' `prob_select_none` for the probability of selecting no arm. -#' \item `rmse`, `rmse_te`: the root mean squared error of the estimates for +#' \item `rmse`, `rmse_te`: the root mean squared errors of the estimates for +#' the selected arm and for the treatment effect, as described in +#' [extract_results()]. +#' \item `mae`, `mae_te`: the median absolute errors of the estimates for #' the selected arm and for the treatment effect, as described in #' [extract_results()]. #' \item `idp`: the ideal design percentage (IDP; 0-100%), see **Details**. @@ -228,7 +231,7 @@ check_performance <- function(object, select_strategy = "control if available", "ratio_ys_p25", "ratio_ys_p75", "ratio_ys_p0", "ratio_ys_p100", "prob_conclusive", "prob_superior", "prob_equivalence", "prob_futility", "prob_max", paste0("prob_select_", c(paste0("arm_", arms), "none")), - "rmse", "rmse_te", "idp"), + "rmse", "rmse_te", "mae", "mae_te", "idp"), est = NA, err_sd = NA, err_mad = NA, lo_ci = NA, hi_ci = NA) # Restrict simulations summarised @@ -255,6 +258,8 @@ check_performance <- function(object, select_strategy = "control if available", mean(is.na(extr_res$selected_arm[restrict_idx])), sqrt(mean(extr_res$sq_err[restrict_idx], na.rm = TRUE)) %f|% NA, sqrt(mean(extr_res$sq_err_te[restrict_idx], na.rm = TRUE)) %f|% NA, + median(abs(extr_res$err[restrict_idx]), na.rm = TRUE) %f|% NA, + median(abs(extr_res$err_te[restrict_idx]), na.rm = TRUE) %f|% NA, calculate_idp(extr_res$selected_arm[restrict_idx], arms, true_ys, highest_is_best) %f|% NA) # Simply object or do bootstrapping @@ -309,6 +314,8 @@ check_performance <- function(object, select_strategy = "control if available", mean(is.na(extr_boot$selected_arm[restrict_idx])), sqrt(mean(extr_boot$sq_err[restrict_idx], na.rm = TRUE)) %f|% NA, sqrt(mean(extr_boot$sq_err_te[restrict_idx], na.rm = TRUE)) %f|% NA, + median(abs(extr_boot$err[restrict_idx]), na.rm = TRUE) %f|% NA, + median(abs(extr_boot$err_te[restrict_idx]), na.rm = TRUE) %f|% NA, calculate_idp(extr_boot$selected_arm[restrict_idx], arms, true_ys, highest_is_best) %f|% NA) } boot_mat diff --git a/R/check_remaining_arms.R b/R/check_remaining_arms.R index 22e489d..4256d61 100644 --- a/R/check_remaining_arms.R +++ b/R/check_remaining_arms.R @@ -6,7 +6,7 @@ #' with a common `control`) across multiple simulated trial results. The #' function supplements the [extract_results()], [check_performance()], and #' [summary()] functions, and is especially useful for designs with `> 2` arms, -#' where it provides details that the other functionality mentioned do not. +#' where it provides details that the other functions mentioned do not. #' #' @param object `trial_results` object, output from the [run_trials()] #' function. diff --git a/R/extract_results.R b/R/extract_results.R index 2b1e494..70d3f5c 100644 --- a/R/extract_results.R +++ b/R/extract_results.R @@ -39,7 +39,9 @@ extract_results_batch <- function(trial_results, final_status = vapply_str(1:n_rep, function(x) trial_results[[x]]$final_status), superior_arm = NA, selected_arm = NA, + err = NA, sq_err = NA, + err_te = NA, sq_err_te = NA, stringsAsFactors = FALSE) @@ -59,7 +61,7 @@ extract_results_batch <- function(trial_results, # Do not consider arms dropped for equivalence before final stop if (cur_status == "equivalence") { # Stopped for equivalence # Only consider equivalent arms declared equivalent at final look - tmp_sel <- tmp_sel[tmp_sel$final_status %in% c("equivalence", "control") & tmp_sel$status_look == df$final_n[[i]], ] + tmp_sel <- tmp_sel[tmp_sel$final_status %in% c("equivalence", "control") & tmp_sel$status_look == trial_results[[i]]$followed_n, ] } else { # Only consider arms not stopped for equivalence tmp_sel <- tmp_sel[tmp_sel$final_status != "equivalence", ] @@ -101,10 +103,12 @@ extract_results_batch <- function(trial_results, selected_index <- which(tmp_res$arms == cur_select) selected_est_y <- tmp_res[[which_ests]][selected_index] selected_true_y <- tmp_res$true_ys[selected_index] + df$err[i] <- selected_est_y - selected_true_y df$sq_err[i] <- (selected_est_y - selected_true_y)^2 if (!is.null(te_comp)){ if (cur_select != te_comp){ te_comp_est_y <- tmp_res[[which_ests]][te_comp_index] + df$err_te[i] <- (selected_est_y - te_comp_est_y) - (selected_true_y - te_comp_true_y) df$sq_err_te[i] <- ( (selected_est_y - te_comp_est_y) - (selected_true_y - te_comp_true_y) )^2 } } @@ -173,18 +177,18 @@ extract_results_batch <- function(trial_results, #' @param te_comp character string, treatment-effect comparator. Can be either #' `NULL` (the default) in which case the **first** `control` arm is used for #' trial designs with a common control arm, or a string naming a single trial -#' `arm`. Will be used when calculating `sq_err_te` (the squared error of the -#' treatment effect comparing the selected arm to the comparator arm, as -#' described below). +#' `arm`. Will be used when calculating `err_te` and `sq_err_te` (the error +#' and the squared error of the treatment effect comparing the selected arm to +#' the comparator arm, as described below). #' @param raw_ests single logical. If `FALSE` (default), the #' posterior estimates (`post_ests` or `post_ests_all`, see [setup_trial()] -#' and [run_trial()]) will be used to calculate `sq_err` (the squared error of -#' the estimated compared to the specified effect in the selected arm) and -#' `sq_err_te` (the squared error of the treatment effect comparing the -#' selected arm to the comparator arm, as described for `te_comp` and below). -#' If `TRUE`, the raw estimates (`raw_ests` or `raw_ests_all`, see -#' [setup_trial()] and [run_trial()]) will be used instead of the posterior -#' estimates. +#' and [run_trial()]) will be used to calculate `err` and `sq_err` (the error +#' and the squared error of the estimated compared to the specified effect in +#' the selected arm) and `err_te` and `sq_err_te` (the error and the squared +#' error of the treatment effect comparing the selected arm to the comparator +#' arm, as described for `te_comp` and below). If `TRUE`, the raw estimates +#' (`raw_ests` or `raw_ests_all`, see [setup_trial()] and [run_trial()]) will +#' be used instead of the posterior estimates. #' @param final_ests single logical. If `TRUE` (recommended) the final estimates #' calculated using outcome data from all patients randomised when trials are #' stopped are used (`post_ests_all` or `raw_ests_all`, see [setup_trial()] @@ -231,17 +235,21 @@ extract_results_batch <- function(trial_results, #' \item `selected_arm`: the final selected arm (as described above). Will #' correspond to the `superior_arm` in simulations stopped for superiority #' and be `NA` if no arm is selected. See `select_strategy` above. +#' \item `err`: the squared error of the estimate in the selected arm, +#' calculated as `estimated effect - true effect` for the selected +#' arm. #' \item `sq_err:` the squared error of the estimate in the selected arm, -#' calculated as `(estimated effect - true effect)^2` for the selected -#' arms. -#' \item `sq_err_te`: the squared error of the treatment effect comparing -#' the selected arm to the comparator arm (as specified in `te_comp`). -#' Calculated as:\cr -#' `((estimated effect in the selected arm - estimated effect in the comparator arm) -` -#' `(true effect in the selected arm - true effect in the comparator arm))^2` \cr -#' Will be `NA` for simulations without a selected arm, with no +#' calculated as `err^2` for the selected arm, with `err` defined above. +#' \item `err_te`: the error of the treatment effect comparing the selected +#' arm to the comparator arm (as specified in `te_comp`). Calculated as:\cr +#' `(estimated effect in the selected arm - estimated effect in the comparator arm) -` +#' `(true effect in the selected arm - true effect in the comparator arm)` +#' \cr Will be `NA` for simulations without a selected arm, with no #' comparator specified (see `te_comp` above), and when the selected arm #' is the comparator arm. +#' \item `sq_err_te`: the squared error of the treatment effect comparing +#' the selected arm to the comparator arm (as specified in `te_comp`), +#' calculated as `err_te^2`, with `err_te` defined above. #' } #' #' @examples diff --git a/R/plot_convergence.R b/R/plot_convergence.R index c1b044a..8282bc9 100644 --- a/R/plot_convergence.R +++ b/R/plot_convergence.R @@ -18,9 +18,9 @@ #' `ratio_ys_p75`, `ratio_ys_p0`, `ratio_ys_p100`, `prob_conclusive`, #' `prob_superior`, `prob_equivalence`, `prob_futility`, `prob_max`, #' `prob_select_*` (with `*` being either "`arm_` for all `arm` names or -#' `none`), `rmse`, `rmse_te`, and `idp`. All may be specified as above, -#' case sensitive, but with either spaces or underlines. Defaults to -#' `"size mean"`. +#' `none`), `rmse`, `rmse_te`, `mae`, `mae_te`, and `idp`. All may be +#' specified as above, case sensitive, but with either spaces or underlines. +#' Defaults to `"size mean"`. #' @param resolution single positive integer, the number of points calculated #' and plotted, defaults to `100` and must be `>= 10`. Higher numbers lead to #' smoother plots, but increases computation time. If the value specified is @@ -98,7 +98,7 @@ plot_convergence <- function(object, metrics = "size mean", resolution = 100, "ratio_ys_p0", "ratio_ys_p100", "prob_conclusive", "prob_superior", "prob_equivalence", "prob_futility", "prob_max", paste0("prob_select_", c(paste0("arm_", arms), "none")), - "rmse", "rmse_te", "idp") + "rmse", "rmse_te", "mae", "mae_te", "idp") # Validate metrics if (!isTRUE(is.character(metrics) & length(metrics) >= 1)) { @@ -187,6 +187,8 @@ plot_convergence <- function(object, metrics = "size mean", resolution = 100, prob_select_none = function(i) mean(is.na(extr_res$selected_arm[start_id:i])) * 100, rmse = function(i) sqrt(mean(extr_res$sq_err[start_id:i], na.rm = TRUE)) %f|% NA, rmse_te = function(i) sqrt(mean(extr_res$sq_err_te[start_id:i], na.rm = TRUE)) %f|% NA, + mae = function(i) median(abs(extr_res$err[start_id:i]), na.rm = TRUE) %f|% NA, + mae_te = function(i) median(abs(extr_res$err_te[start_id:i]), na.rm = TRUE) %f|% NA, idp = function(i) calculate_idp(extr_res$selected_arm[start_id:i], arms, true_ys, highest_is_best) %f|% NA ) } @@ -221,6 +223,8 @@ plot_convergence <- function(object, metrics = "size mean", resolution = 100, metric_labels[metric_labels == "Idp"] <- "IDP (%)" metric_labels[metric_labels == "Rmse"] <- "RMSE" metric_labels[metric_labels == "Rmse te"] <- "RMSE TE" + metric_labels[metric_labels == "Mae"] <- "MAE" + metric_labels[metric_labels == "Mae te"] <- "MAE TE" metric_labels <- gsub("select ", "", metric_labels) plot_dta$labels <- factor(vapply_str(plot_dta$metric, function(l) metric_labels[which(l == valid_metrics)]), levels = vapply_str(metrics, function(l) metric_labels[which(l == valid_metrics)])) @@ -238,7 +242,7 @@ plot_convergence <- function(object, metrics = "size mean", resolution = 100, ggplot2::scale_y_continuous(name = plot_dta$labels[1]) } else { # Multiple metrics plotted if (is.null(nrow) & is.null(ncol)) { # Set nrow if both nrow and ncol are NULL - nrow <- ceiling(sqrt(length(metrics))) + nrow <- ifelse(length(metrics) <= 3, length(metrics), ceiling(sqrt(length(metrics)))) } p <- p + ggplot2::scale_y_continuous(name = NULL) + diff --git a/R/plot_metrics_ecdf.R b/R/plot_metrics_ecdf.R index f482d91..57ad5d9 100644 --- a/R/plot_metrics_ecdf.R +++ b/R/plot_metrics_ecdf.R @@ -9,13 +9,21 @@ #' @inheritParams check_performance #' @param metrics the performance metrics to plot, as described in #' [extract_results()]. Multiple metrics may be plotted at the same time. -#' Valid metrics include: `size`, `sum_ys`, and `ratio_ys_mean`. All may be -#' specified using either spaces or underlines (case sensitive). Defaults to -#' plotting all three. +#' Valid metrics include: `size`, `sum_ys`, `ratio_ys_mean`, `sq_err`, +#' `sq_err_te`, `err`, `err_te`, `abs_err`, `abs_err_te`, (as described in +#' [extract_results()], with the addition of `abs_err` and `abs_err_te`, which +#' are the absolute errors, i.e., `abs(err)` and `abs(err_te)`). All +#' may be specified using either spaces or underlines (case sensitive). +#' Defaults to plotting `size`, `sum_ys`, and `ratio_ys_mean`. #' @param nrow,ncol the number of rows and columns when plotting multiple #' metrics in the same plot (using faceting in `ggplot2`). Defaults to `NULL`, #' in which case this will be determined automatically. #' +#' @details +#' +#' Note that the arguments related to arm selection and error calculation are +#' only relevant if errors are visualised. +#' #' @return A `ggplot2` plot object. #' #' @export @@ -47,6 +55,9 @@ #' [plot_convergence()], [check_remaining_arms()]. #' plot_metrics_ecdf <- function(object, metrics = c("size", "sum_ys", "ratio_ys"), + select_strategy = "control if available", + select_last_arm = FALSE, select_preferences = NULL, + te_comp = NULL, raw_ests = FALSE, final_ests = NULL, restrict = NULL, nrow = NULL, ncol = NULL, cores = NULL) { @@ -59,7 +70,8 @@ plot_metrics_ecdf <- function(object, metrics = c("size", "sum_ys", "ratio_ys"), "to plot, and each metric must be specified only once.") } else { metrics <- chartr("_", " ", metrics) # Replace underlines with spaces - if (!all(metrics %in% c("size", "sum ys", "ratio ys"))) { + if (!all(metrics %in% c("size", "sum ys", "ratio ys", "sq err", "sq err te", + "err", "err te", "abs err", "abs err te"))) { stop0("Invalid metric(s) specified. Type 'help(plot_metrics_ecdf)' to see a list of the metrics ", "that may be specified.") } @@ -76,13 +88,21 @@ plot_metrics_ecdf <- function(object, metrics = c("size", "sum_ys", "ratio_ys"), } # Extract results and values from trial specification object - extr_res <- extract_results(object, cores = cores) # Other arguments not used + extr_res <- extract_results(object, select_strategy = select_strategy, + select_last_arm = select_last_arm, + select_preferences = select_preferences, + te_comp = te_comp, raw_ests = raw_ests, + final_ests = final_ests, cores = cores) if (isTRUE(restrict == "superior")) { extr_res <- extr_res[!is.na(extr_res$superior_arm), ] } else if (isTRUE(restrict == "selected")) { extr_res <- extr_res[!is.na(extr_res$selected_arm), ] } + if (nrow(extr_res) <= 1) { + stop0("Empirical cumulative distributions cannot be plotted as the number of simulation ", + "results to plot is <= 1", ifelse(is.null(restrict), ".", " after restriction.")) + } # Prepare data for plotting plot_dta <- data.frame(metric = character(0), value = numeric(0)) @@ -101,7 +121,53 @@ plot_metrics_ecdf <- function(object, metrics = c("size", "sum_ys", "ratio_ys"), data.frame(metric = "Ratio ys", value = extr_res$ratio_ys)) } + if ("sq err" %in% metrics) { + plot_dta <- rbind(plot_dta, + data.frame(metric = "Error^2", + value = extr_res$sq_err)) + } + if ("sq err te" %in% metrics) { + if (sum(!is.na(extr_res$sq_err_te)) <= 1) { + stop0("Empirical cumulative distribution of sq_err_te cannot be plotted as the number of non-NA values is <= 1.") + } + plot_dta <- rbind(plot_dta, + data.frame(metric = "Error TE^2", + value = extr_res$sq_err_te)) + } + if ("err" %in% metrics) { + plot_dta <- rbind(plot_dta, + data.frame(metric = "Error", + value = extr_res$err)) + } + if ("err te" %in% metrics) { + if (sum(!is.na(extr_res$err_te)) <= 1) { + stop0("Empirical cumulative distribution of err_te can not be plotted as the number of non-NA values is <= 1.") + } + plot_dta <- rbind(plot_dta, + data.frame(metric = "Error TE", + value = extr_res$err_te)) + } + if ("abs err" %in% metrics) { + plot_dta <- rbind(plot_dta, + data.frame(metric = "Abs(error)", + value = abs(extr_res$err))) + } + if ("abs err te" %in% metrics) { + if (sum(!is.na(extr_res$err_te)) <= 1) { + stop0("Empirical cumulative distribution of abs_err_te can not be plotted as the number of non-NA values is <= 1.") + } + plot_dta <- rbind(plot_dta, + data.frame(metric = "Abs(error TE)", + value = abs(extr_res$err_te))) + } + plot_dta <- na.omit(plot_dta) # Ignore NAs for error metrics substr(metrics, 1, 1) <- toupper(substr(metrics, 1, 1)) + metrics[metrics == "Sq err"] <- "Error^2" + metrics[metrics == "Sq err te"] <- "Error TE^2" + metrics[metrics == "Err"] <- "Error" + metrics[metrics == "Err te"] <- "Error TE" + metrics[metrics == "Abs err"] <- "Abs(error)" + metrics[metrics == "Abs err te"] <- "Abs(error TE)" plot_dta <- transform(plot_dta, metric = factor(metric, levels = metrics)) @@ -116,7 +182,7 @@ plot_metrics_ecdf <- function(object, metrics = c("size", "sum_ys", "ratio_ys"), ggplot2::scale_y_continuous(name = metrics, breaks = 0:5 * 0.2, labels = paste0(0:5 * 20, "%"), limits = 0:1, expand = c(0, 0)) } else { # Multiple metrics plotted if (is.null(nrow) & is.null(ncol)) { # Set nrow if both nrow and ncol are NULL - nrow <- length(metrics) + nrow <- ifelse(length(metrics) <= 3, length(metrics), ceiling(sqrt(length(metrics)))) } p <- p + ggplot2::scale_y_continuous(name = NULL, breaks = 0:5 * 0.2, labels = paste0(0:5 * 20, "%"), limits = 0:1, expand = c(0, 0)) + diff --git a/R/plot_status.R b/R/plot_status.R index c8c635f..48c4a61 100644 --- a/R/plot_status.R +++ b/R/plot_status.R @@ -110,7 +110,7 @@ plot_status.trial_results <- function(object, x_value = "look", arm = NULL, # Facet if plotting one or more arms (to add arm labels to the top) if (!arm_null) { if (is.null(nrow) & is.null(ncol)) { # Set nrow if both nrow and ncol are NULL - nrow <- ceiling(sqrt(length(arm))) + nrow <- ifelse(length(arm) <= 3, length(arm), ceiling(sqrt(length(arm)))) } p <- p + ggplot2::facet_wrap(ggplot2::vars(arm_facet), scales = "free_x", nrow = nrow, ncol = ncol, strip.position = "top") + diff --git a/R/print.R b/R/print.R index dc0d67a..eac1fd6 100644 --- a/R/print.R +++ b/R/print.R @@ -76,8 +76,14 @@ print.trial_spec <- function(x, prob_digits = 3, ...) { cat("* Best arms:", paste0(x$best_arm, collapse = " and ")) } - cat0("\n\nArms, true outcomes, starting allocation probabilities \n", - "and allocation probability limits:\n") + cat0(paste0("\n\nArms, true outcomes, starting allocation probabilities \n", + "and allocation probability limits", + ifelse(is.null(x$rescale_probs), "", c( + fixed = " (fixed_probs rescaled)", + limits = " (min/max_probs rescaled)", + both = " (fixed/min/max_probs rescaled)" + )[x$rescale_probs]), + ":\n")) print(x$trial_arms, digits = prob_digits, row.names = FALSE) # Samples size and looks @@ -362,8 +368,9 @@ print.trial_results_summary <- function(x, digits = 1, ...) { "* Selection probabilities: ", paste0(vapply_str(which(substr(names(x), 1, 12) == "prob_select_"), function(i){ paste0(ifelse(names(x)[i] == "prob_select_none", "None", substr(names(x)[i], 17, nchar(names(x)[i]))), ": ", fmt_dig(x[[names(x)[i]]]*100, digits), "%")}), collapse = " | "), "\n", - "* RMSE: ", fmt_dig(x$rmse, 5), "\n", - "* RMSE treatment effect: ", ifelse(is.na(x$rmse_te), "not estimated", fmt_dig(x$rmse_te, 5)), "\n", + "* RMSE / MAE: ", fmt_dig(x$rmse, 5), " / ", fmt_dig(x$mae, 5), "\n", + "* RMSE / MAE treatment effect: ", ifelse(is.na(x$rmse_te), "not estimated", fmt_dig(x$rmse_te, 5)), + " / ", ifelse(is.na(x$mae_te), "not estimated", fmt_dig(x$mae_te, 5)), "\n", "* Ideal design percentage: ", ifelse(is.na(x$idp), "not estimable", paste0(fmt_dig(x$idp, digits), "%")), # Technical simulation details diff --git a/R/prob_funs.R b/R/prob_funs.R index 4b79acf..29c96ee 100644 --- a/R/prob_funs.R +++ b/R/prob_funs.R @@ -124,6 +124,14 @@ prob_all_equi <- function(m, equivalence_diff = NULL) { #' @param match_arm index of the `control` arm. If not `NULL` (default), the #' control arm allocation probability will be similar to that of the best #' non-control arm. Must be `NULL` in designs without a common control arm. +#' @param rescale_fixed logical indicating whether `fixed_probs` should be +#' rescaled following arm dropping. +#' @param rescale_limits logical indicating whether `min/max_probs` should be +#' rescaled following arm dropping. +#' @param rescale_factor numerical, rescale factor defined as +#' `initial number of arms/number of active arms`. +#' @param rescale_ignore `NULL` or index of an arm that will be ignored by the +#' `rescale_fixed` and `rescale_limits` arguments. #' #' @return A named (according to the `arms`) numeric vector with updated #' allocation probabilities. @@ -133,7 +141,9 @@ prob_all_equi <- function(m, equivalence_diff = NULL) { #' @keywords internal #' reallocate_probs <- function(probs_best, fixed_probs, min_probs, max_probs, - soften_power = 1, match_arm = NULL) { + soften_power = 1, match_arm = NULL, + rescale_fixed = FALSE, rescale_limits = FALSE, + rescale_factor = 1, rescale_ignore = NULL) { # Match the control arm allocation ratio to the best arm's ratio if specified if (!is.null(match_arm) & length(probs_best) > 1) { @@ -151,6 +161,18 @@ reallocate_probs <- function(probs_best, fixed_probs, min_probs, max_probs, return(setNames(rescale(probs_best^soften_power), names(probs_best))) } + # Rescale fixed_probs, min_probs, and max_probs + if ((rescale_factor - 1) > .Machine$double.eps^0.5) { # Ignore if unnecessary + rescale_idx <- which(!(1:length(probs_best) %in% rescale_ignore)) + if (rescale_fixed & length(rescale_idx) > 0) { + fixed_probs[rescale_idx] <- fixed_probs[rescale_idx] * rescale_factor + } + if (rescale_limits & length(rescale_idx) > 0) { + min_probs[rescale_idx] <- min_probs[rescale_idx] * rescale_factor + max_probs[rescale_idx] <- 1 - ( (1 - max_probs[rescale_idx]) * rescale_factor) + } + } + # If all probabilities are fixed, just return those unless some arms are # dropped (or it exceeds 1 due to a fixed control arm ratio), then rescale. # Does not consider matching if all are fixed. diff --git a/R/run_trial.R b/R/run_trial.R index ca96ca8..942f6e3 100644 --- a/R/run_trial.R +++ b/R/run_trial.R @@ -193,6 +193,8 @@ run_trial <- function(trial_spec, seed = NULL, sparse = FALSE) { # Prepare variables/extract from specification trial_arms <- as.list(trial_spec$trial_arms) arms <- trial_arms$arms + rescale_fixed <- isTRUE(trial_spec$rescale_probs %in% c("fixed", "both")) + rescale_limits <- isTRUE(trial_spec$rescale_probs %in% c("limits", "both")) control <- trial_spec$control control_prob_fixed <- trial_spec$control_prob_fixed match_arm <- isTRUE(control_prob_fixed == "match") # Match control arm allocation probability to highest non-control arm @@ -387,7 +389,10 @@ run_trial <- function(trial_spec, seed = NULL, sparse = FALSE) { fixed_probs = trial_arms$fixed_probs[aai], min_probs = trial_arms$min_probs[aai], max_probs = trial_arms$max_probs[aai], - soften_power = soften_power[look] + soften_power = soften_power[look], + rescale_fixed = rescale_fixed, + rescale_limits = rescale_limits, + rescale_factor = n_arms/length(aai) ) } @@ -596,7 +601,11 @@ run_trial <- function(trial_spec, seed = NULL, sparse = FALSE) { min_probs = min_probs_control, max_probs = max_probs_control, soften_power = soften_power[look], - if (match_arm) active_control_arm else NULL + match_arm = if (match_arm) active_control_arm else NULL, + rescale_fixed = rescale_fixed, + rescale_limits = rescale_limits, + rescale_factor = n_arms/length(aai), + rescale_ignore = if (!is.null(control_prob_fixed)) which(trial_arms$arms[aai] == control) else NULL ) } diff --git a/R/setup_trial.R b/R/setup_trial.R index dc64a0b..0438276 100644 --- a/R/setup_trial.R +++ b/R/setup_trial.R @@ -14,6 +14,7 @@ validate_trial <- function(arms, true_ys, start_probs = NULL, fixed_probs = NULL, min_probs = rep(NA, length(arms)), max_probs = rep(NA, length(arms)), + rescale_probs = NULL, data_looks = NULL, max_n = NULL, look_after_every = NULL, randomised_at_looks = NULL, @@ -40,14 +41,17 @@ validate_trial <- function(arms, true_ys, start_probs = NULL, if (length(unique(arms)) != length(arms)) { stop0("All arms must have unique names.") } + if (n_arms < 2) { + stop0("Two or more arms required.") + } # If control_prob_fixed is set to a valid non-numeric value, then set according to that + control_prob_fixed_orig <- control_prob_fixed # Save original before editing if (!is.null(control_prob_fixed)) { if (is.null(control) | sum(control %in% arms) != 1) { stop0("control_prob_fixed is specified, but no single valid control is specified.") } if (isTRUE(length(control_prob_fixed) == 1 & control_prob_fixed %in% c("sqrt-based", "sqrt-based start", "sqrt-based fixed"))) { - control_prob_fixed_orig <- control_prob_fixed if (!is.null(start_probs)) { stop0("When control_prob_fixed is set to 'sqrt-based', 'sqrt-based start', ", "or 'sqrt-based fixed', start_probs must be NULL.") @@ -164,6 +168,32 @@ validate_trial <- function(arms, true_ys, start_probs = NULL, stop0("fixed_probs or max_probs specified for all arms, but they sum to less than 1.") } + # Check validity of rescale_probs + if (!is.null(rescale_probs)) { + # Check validity of value + if (isTRUE(length(rescale_probs) == 1 & rescale_probs %in% c("fixed", "limits", "both"))) { + # Must be null for two-arm trials or if control_prob_fixed is "sqrt-based fixed" + if (n_arms == 2) { + stop0("rescale_probs must be NULL for trial specifications with 2 arms.") + } else if (isTRUE(control_prob_fixed_orig == "sqrt-based fixed")) { + stop0("rescale_probs must be NULL when control_prob_fixed is set to 'sqrt-based fixed'.") + } + # Valid values to be rescaled must be provided + if (rescale_probs %in% c("fixed", "both")) { + if (sum(!is.na(fixed_probs)) <= 0 + isTRUE(control_prob_fixed_orig %in% c("sqrt-based", "sqrt-based start")) ) { + stop0("rescale_probs is '", rescale_probs, "' but no fixed_probs that can be rescaled are specified.") + } + } + if (rescale_probs %in% c("limits", "both")) { + if (sum(!is.na(min_probs), !is.na(max_probs)) == 0) { + stop0("rescale_probs is '", rescale_probs, "' but no min_probs or max_probs that can be rescaled are specified.") + } + } + } else { + stop0("rescale_probs must be either NULL or 'fixed', 'limits' or 'both'.") + } + } + # Check or setup data looks if (!is.null(data_looks)) { # data_looks is specified, validate that n_data_looks <- length(data_looks) @@ -232,6 +262,10 @@ validate_trial <- function(arms, true_ys, start_probs = NULL, stop0("When control_prob_fixed is specified and is not 'match', fixed_probs for ", "the control arm must be set to the first value of control_prob_fixed.") } + } else { + if (!is.na(fixed_probs[control_index])) { + stop0("When control_prob_fixed is 'match', no fixed_probs can be specified for the control arm.") + } } } } @@ -407,6 +441,7 @@ validate_trial <- function(arms, true_ys, start_probs = NULL, trial_arms <- data.frame(arms, true_ys, start_probs, fixed_probs, min_probs, max_probs, stringsAsFactors = FALSE) structure(list(trial_arms = trial_arms, + rescale_probs = rescale_probs, data_looks = data_looks, max_n = max_n, look_after_every = look_after_every, @@ -482,6 +517,29 @@ validate_trial <- function(arms, true_ys, start_probs = NULL, #' probabilities; higher probabilities will be rounded down to these values. #' Must be `NA` (default for all arms) if no threshold is wanted and for arms #' using fixed allocation probabilities. +#' @param rescale_probs `NULL` (default) or one of either `"fixed"`, `"limits"`, +#' or `"both"`. Rescales `fixed_probs` (if `"fixed"` or `"both"`) and +#' `min_probs/max_probs` (if `"limits"` or `"both"`) after arm dropping in +#' trial specifications with `>2 arms` using a `rescale_factor` defined as +#' `initial number of arms/number of active arms`. `"fixed_probs` and +#' `min_probs` are rescaled as `initial value * rescale factor`, except for +#' `fixed_probs` controlled by the `control_prob_fixed` argument, which are +#' never rescaled. `max_probs` are rescaled as +#' `1 - ( (1 - initial value) * rescale_factor)`.\cr +#' Must be `NULL` if there are only `2 arms` or if `control_prob_fixed` is +#' `"sqrt-based fixed"`. If not `NULL`, one or more valid non-`NA` values must +#' be specified for either `min_probs/max_probs` or `fixed_probs` (not +#' counting a fixed value for the original `control` if `control_prob_fixed` +#' is `"sqrt-based"/"sqrt-based start"/"sqrt-based fixed"`).\cr +#' **Note:** using this argument and specific combinations of values in +#' the other arguments may lead to invalid combined (total) allocation +#' probabilities after arm dropping, in which case all probabilities will +#' ultimately be rescaled to sum to `1`. It is the responsibility of the user +#' to ensure that rescaling fixed allocation probabilities and minimum/maximum +#' allocation probability limits will not lead to invalid or unexpected +#' allocation probabilities after arm dropping.\cr +#' Finally, any initial values that are overwritten by the +#' `control_prob_fixed` argument after arm dropping will not be rescaled. #' @param data_looks vector of increasing integers, specifies when to conduct #' adaptive analyses (= the total number of patients with available outcome #' data at each adaptive analysis). The last number in the vector represents @@ -522,10 +580,11 @@ validate_trial <- function(arms, true_ys, start_probs = NULL, #' be set `NULL` (the default), in which case the control arm allocation #' probability will not be fixed if control arms change (the allocation #' probability for the first control arm may still be fixed using -#' `fixed_probs`). If not `NULL`, a vector of probabilities of either length -#' `1` or `number of arms - 1` can be provided, or one of the special -#' arguments `"sqrt-based"`, `"sqrt-based start"`, `"sqrt-based fixed"` or -#' `"match"`. See [setup_trial()] **Details** for details on how this affects +#' `fixed_probs`, but will not be 'reused' for the new control arm).\cr +#' If not `NULL`, a vector of probabilities of either length `1` or +#' `number of arms - 1` can be provided, or one of the special arguments +#' `"sqrt-based"`, `"sqrt-based start"`, `"sqrt-based fixed"` or `"match"`.\cr +#' See [setup_trial()] **Details** for details on how this affects #' trial behaviour. #' @param inferiority single numeric value or vector of numeric values of the #' same length as the maximum number of possible adaptive analyses, specifying @@ -708,8 +767,8 @@ validate_trial <- function(arms, true_ys, start_probs = NULL, #' user-specified functions (or uses objects defined by the user outside these #' functions or the [setup_trial()]-call) or functions from external packages #' and simulations are conducted on multiple cores, these objects or functions -#' must be exported or prefixed with their namespaces, respectively, as -#' described in [setup_cluster()] and [run_trials()]. +#' must be prefixed with their namespaces (i.e., `package::function()`) or +#' exported, as described in [setup_cluster()] and [run_trials()]. #' #' \strong{More information on arguments} #' - `control`: if one or more treatment arms are superior to the control arm @@ -739,14 +798,17 @@ validate_trial <- function(arms, true_ys, start_probs = NULL, #' scaled to sum to 1, which will generally increase power for comparisons #' against the common `control`, as discussed in, e.g., *Park et al, 2020* #' \doi{10.1016/j.jclinepi.2020.04.025}.\cr -#' If `"sqrt-based"`, square-root-transformation-based allocation probabilities -#' will also be used for new controls when arms are dropped. If -#' `"sqrt-based start"`, the control arm will be fixed to this allocation -#' probability at all times (also after arm dropping, with rescaling as -#' necessary, as specified above). If `"sqrt-based fixed"` is chosen, -#' square-root-transformation-based allocation probabilities will be used and -#' all allocation probabilities will be fixed throughout the trial (with -#' rescaling when arms are dropped).\cr +#' If `"sqrt-based"` or `"sqrt-based fixed"`, square-root-transformation-based +#' allocation probabilities will be used initially and also for new controls +#' when arms are dropped (with probabilities always calculated based on the +#' number of active non-control arms). If `"sqrt-based"`, response-adaptive +#' randomisation will be used for non-control arms, while the non-control arms +#' will use fixed, square-root based allocation probabilities at all times (with +#' probabilities always calculated based on the number of active non-control +#' arms). If `"sqrt-based start"`, the control arm allocation probability will +#' be fixed to a square-root based probability at all times calculated according +#' to the initial number of arms (with this probability also being used for new +#' control(s) when the original control is dropped).\cr #' If `"match"` is specified, the control group allocation probability will #' always be *matched* to be similar to the highest non-control arm allocation #' probability. @@ -892,6 +954,7 @@ setup_trial <- function(arms, true_ys, fun_y_gen = NULL, fun_draws = NULL, start_probs = NULL, fixed_probs = NULL, min_probs = rep(NA, length(arms)), max_probs = rep(NA, length(arms)), + rescale_probs = NULL, data_looks = NULL, max_n = NULL, look_after_every = NULL, randomised_at_looks = NULL, @@ -906,8 +969,9 @@ setup_trial <- function(arms, true_ys, fun_y_gen = NULL, fun_draws = NULL, add_info = NULL) { validate_trial(arms = arms, true_ys = true_ys, start_probs = start_probs, fixed_probs = fixed_probs, - min_probs = min_probs, max_probs = max_probs, data_looks = data_looks, max_n = max_n, - look_after_every = look_after_every, randomised_at_looks = randomised_at_looks, + min_probs = min_probs, max_probs = max_probs, rescale_probs = rescale_probs, + data_looks = data_looks, max_n = max_n, look_after_every = look_after_every, + randomised_at_looks = randomised_at_looks, control = control, control_prob_fixed = control_prob_fixed, inferiority = inferiority, superiority = superiority, equivalence_prob = equivalence_prob, equivalence_diff = equivalence_diff, equivalence_only_first = equivalence_only_first, @@ -974,6 +1038,7 @@ setup_trial_binom <- function(arms, true_ys, start_probs = NULL, fixed_probs = NULL, min_probs = rep(NA, length(arms)), max_probs = rep(NA, length(arms)), + rescale_probs = NULL, data_looks = NULL, max_n = NULL, look_after_every = NULL, randomised_at_looks = NULL, @@ -988,6 +1053,7 @@ setup_trial_binom <- function(arms, true_ys, start_probs = NULL, description = "generic binomially distributed outcome trial") { # Validate specific arguments to trials with binary outcomes + if (missing(true_ys)) true_ys <- NULL # to avoid incorrect error if (!isFALSE(length(arms) != length(true_ys) | any(is.na(true_ys)) | any(true_ys > 1) | any(true_ys < 0) | !is.numeric(true_ys))) { stop0("true_ys must be a vector of the same length as the number of arms containing ", "values (event probabilities) between 0 and 1 with no missing values.") @@ -995,8 +1061,9 @@ setup_trial_binom <- function(arms, true_ys, start_probs = NULL, # General setup and validation trial <- validate_trial(arms = arms, true_ys = true_ys, start_probs = start_probs, fixed_probs = fixed_probs, - min_probs = min_probs, max_probs = max_probs, data_looks = data_looks, max_n = max_n, - look_after_every = look_after_every, randomised_at_looks = randomised_at_looks, + min_probs = min_probs, max_probs = max_probs, rescale_probs = rescale_probs, + data_looks = data_looks, max_n = max_n, look_after_every = look_after_every, + randomised_at_looks = randomised_at_looks, control = control, control_prob_fixed = control_prob_fixed, inferiority = inferiority, superiority = superiority, equivalence_prob = equivalence_prob, equivalence_diff = equivalence_diff, equivalence_only_first = equivalence_only_first, @@ -1091,6 +1158,7 @@ setup_trial_norm <- function(arms, true_ys, sds, start_probs = NULL, fixed_probs = NULL, min_probs = rep(NA, length(arms)), max_probs = rep(NA, length(arms)), + rescale_probs = NULL, data_looks = NULL, max_n = NULL, look_after_every = NULL, randomised_at_looks = NULL, @@ -1105,6 +1173,8 @@ setup_trial_norm <- function(arms, true_ys, sds, start_probs = NULL, description = "generic normally distributed outcome trial") { # Validate specific arguments to generic continuous, normally distributed outcome trials + if (missing(true_ys)) true_ys <- NULL # to avoid incorrect error + if (missing(sds)) sds <- NULL # to avoid incorrect error if (!isFALSE(length(arms) != length(true_ys) | any(is.na(true_ys)) | !is.numeric(true_ys) | length(arms) != length(sds) | any(is.na(sds)) | !is.numeric(sds) | any(sds <= 0))) { stop0("true_ys and sds must be vectors of the same length as the number of arms and all sds must be > 0.") @@ -1112,8 +1182,9 @@ setup_trial_norm <- function(arms, true_ys, sds, start_probs = NULL, # General setup and validation and return validate_trial(arms = arms, true_ys = true_ys, start_probs = start_probs, fixed_probs = fixed_probs, - min_probs = min_probs, max_probs = max_probs, data_looks = data_looks, max_n = max_n, - look_after_every = look_after_every, randomised_at_looks = randomised_at_looks, + min_probs = min_probs, max_probs = max_probs, rescale_probs = rescale_probs, + data_looks = data_looks, max_n = max_n, look_after_every = look_after_every, + randomised_at_looks = randomised_at_looks, control = control, control_prob_fixed = control_prob_fixed, inferiority = inferiority, superiority = superiority, equivalence_prob = equivalence_prob, equivalence_diff = equivalence_diff, equivalence_only_first = equivalence_only_first, diff --git a/R/summary.R b/R/summary.R index 3d42195..931c830 100644 --- a/R/summary.R +++ b/R/summary.R @@ -23,11 +23,11 @@ #' `ratio_ys_p0`, `ratio_ys_p100`, `prob_conclusive`, `prob_superior`, #' `prob_equivalence`, `prob_futility`, `prob_max`, `prob_select_*` (with #' `*` being either "`arm_` for all `arm` names or `none`), `rmse`, -#' `rmse_te`, and `idp`: performance metrics as described in -#' [check_performance()]. Note that all `sum_ys_` and `ratio_ys_` measures -#' use outcome data from all randomised patients, regardless of whether they -#' had outcome data available at the last analysis or not, as described in -#' [extract_results()]. +#' `rmse_te`, `mae`, `mae_te`, and `idp`: performance metrics as described +#' in [check_performance()]. Note that all `sum_ys_` and `ratio_ys_` +#' measures use outcome data from all randomised patients, regardless of +#' whether they had outcome data available at the last analysis or not, as +#' described in [extract_results()]. #' \item `select_strategy`, `select_last_arm`, `select_preferences`, #' `te_comp`, `raw_ests`, `final_ests`, `restrict`: as specified above. #' \item `control`: the control arm specified by [setup_trial()], diff --git a/R/update_saved_trials.R b/R/update_saved.R similarity index 53% rename from R/update_saved_trials.R rename to R/update_saved.R index 3f9922d..ec949a6 100644 --- a/R/update_saved_trials.R +++ b/R/update_saved.R @@ -36,12 +36,14 @@ #' The following changes are made according to the version of `adaptr` used to #' generate the original `"trial_results"` object: #' \itemize{ -#' \item `v1.2.0+`: only updates the version number. +#' \item `v1.2.0+`: updates version number and the `reallocate_probs` +#' argument in the embedded trial specification. #' \item `v1.1.1 or earlier`: updates version number and everything related #' to follow-up and data collection lag (in these versions, the #' `randomised_at_looks` argument in the [setup_trial()] functions did not #' exist, but for practical purposes was identical to the number of -#' patients with available data at each look). +#' patients with available data at each look) and the `reallocate_probs` +#' argument in the embedded trial specification. #' } #' #' @return Invisibly returns the updated `"trial_results"`-object. @@ -69,7 +71,8 @@ update_saved_trials <- function(path, version = NULL, compress = TRUE) { # Update the trial_spec-part of the object, re-arrange order of objects, set class object$trial_spec$randomised_at_looks <- object$trial_spec$data_looks - object$trial_spec <- object$trial_spec[c("trial_arms", "data_looks", "max_n", "look_after_every", + object$trial_spec <- c(object$trial_spec, list(rescale_probs = NULL)) + object$trial_spec <- object$trial_spec[c("trial_arms", "rescale_probs", "data_looks", "max_n", "look_after_every", "n_data_looks", "randomised_at_looks", "control", "control_prob_fixed", "inferiority", "superiority", "equivalence_prob", "equivalence_diff", "equivalence_only_first", "futility_prob", "futility_diff", "futility_only_first", @@ -119,7 +122,17 @@ update_saved_trials <- function(path, version = NULL, compress = TRUE) { class(object$trial_results[[i]]) <- c("trial_result", "list") } } else if (.adaptr_version >= "1.2.0") { - # Only update the version number in the saved object + # Update the trial_spec-part of the object, re-arrange order of objects, set class + object$trial_spec <- c(object$trial_spec, list(rescale_probs = NULL)) + object$trial_spec <- object$trial_spec[c("trial_arms", "rescale_probs", "data_looks", "max_n", "look_after_every", + "n_data_looks", "randomised_at_looks", "control", "control_prob_fixed", + "inferiority", "superiority", "equivalence_prob", "equivalence_diff", + "equivalence_only_first", "futility_prob", "futility_diff", "futility_only_first", + "highest_is_best", "soften_power", "best_arm", "cri_width", "n_draws", "robust", + "description", "add_info", "fun_y_gen", "fun_draws", "fun_raw_est")] + class(object$trial_spec) <- c("trial_spec", "list") + + # Updated the version number object$adaptr_version <- .adaptr_version } # Save and return invisibly @@ -128,3 +141,97 @@ update_saved_trials <- function(path, version = NULL, compress = TRUE) { } invisible(object) } + + + +#' Update previously saved calibration result +#' +#' This function updates a previously saved `"trial_calibration"`-object created +#' and saved by [calibrate_trial()] using a previous version of `adaptr`, +#' including the embedded trial specification and trial results objects +#' (internally using the [update_saved_trials()] function). This allows the +#' use of calibration results, including the calibrated trial specification and +#' the best simulations results from the calibration process, to be used without +#' errors by this version of the package. The function should be run only once +#' per saved simulation object and will issue a warning if the object is already +#' up to date. And overview of the changes made according to the `adaptr` package +#' version used to generate the original object is provided in **Details**.\cr +#' +#' @param path single character; the path to the saved +#' `"trial_calibration"`-object containing the calibration result saved by +#' [calibrate_trial()]. +#' @param version passed to [saveRDS()] when saving the updated object, defaults +#' to `NULL` (as in [saveRDS()]), which means that the current default version +#' is used. +#' @param compress passed to [saveRDS()] when saving the updated object, +#' defaults to `TRUE` (as in [saveRDS()]), see [saveRDS()] for other options. +#' +#' @details +#' +#' The following changes are made according to the version of `adaptr` used to +#' generate the original `"trial_calibration"` object: +#' \itemize{ +#' \item `v1.3.0+`: updates version number of the +#' `"trial_calibration"`-object and updates the embedded +#' `"trial_results"`-object (saved in `$best_sims`, if any) and +#' `"trial_spec"`-objects (saved in `$input_trial_spec` and +#' `$best_trial_spec`) as described in [update_saved_trials()]. +#' } +#' +#' @return Invisibly returns the updated `"trial_calibration"`-object. +#' +#' @export +#' +#' @seealso +#' [run_trials()]. +#' +update_saved_calibration <- function(path, version = NULL, compress = TRUE) { + # Check if file exists at path + if (!file.exists(path)) stop0("path must be a valid path to a trial_calibration-object.") + object <- readRDS(path) + if (!inherits(object, "trial_calibration")) { + stop0("path must lead to a valid trial_calibration-object previously saved by calibrate_trial().") + } + prev_version <- object$adaptr_version + save_object <- TRUE + if (isTRUE(!is.null(prev_version) & prev_version == .adaptr_version)) { # Already up-to-date + save_object <- FALSE + warning0("path leads to a trial_calibration-object that is already up to date; object not updated.") + } else { # Currently, the same is done for all previous versions - update later if required + + # Update overall object + object$adaptr_version <- .adaptr_version + + # Update the input and updated trial specifications + # Only contents changed after calibration introduced updated + if (!"rescale_probs" %in% names(object$input_trial_spec)) { # if rescale_probs is missing, add to both + object$input_trial_spec <- c(object$input_trial_spec, list(rescale_probs = NULL)) + object$best_trial_spec <- c(object$best_trial_spec, list(rescale_probs = NULL)) + list_order <- c("trial_arms", "rescale_probs", "data_looks", "max_n", "look_after_every", + "n_data_looks", "randomised_at_looks", "control", "control_prob_fixed", + "inferiority", "superiority", "equivalence_prob", "equivalence_diff", + "equivalence_only_first", "futility_prob", "futility_diff", "futility_only_first", + "highest_is_best", "soften_power", "best_arm", "cri_width", "n_draws", "robust", + "description", "add_info", "fun_y_gen", "fun_draws", "fun_raw_est") + object$input_trial_spec <- object$input_trial_spec[list_order] + class(object$input_trial_spec) <- c("trial_spec", "list") + object$best_trial_spec <- object$best_trial_spec[list_order] + class(object$best_trial_spec) <- c("trial_spec", "list") + } + + # Updated a saved results object if any (not saved if new simulations not necessary for calibration) + if (!is.null(object$best_sims)) { + # Save in a temporary file to allow updated_saved_trials to update it + tmp_file <- tempfile() + on.exit(try(file.remove(tmp_file)), add = TRUE, after = FALSE) + saveRDS(object$best_sims, file = tmp_file, version = version, compress = compress) + object$best_sims <- update_saved_trials(tmp_file, version = version, compress = compress) + } + + } + # Save and return invisibly + if (save_object) { + saveRDS(object, file = path, version = version, compress = compress) + } + invisible(object) +} diff --git a/README.Rmd b/README.Rmd index 18995c9..7578868 100644 --- a/README.Rmd +++ b/README.Rmd @@ -28,9 +28,8 @@ using adaptive stopping, adaptive arm dropping and/or response-adaptive randomisation. The package has been developed as part of the -[INCEPT (Intensive Care Platform Trial) project](https://incept.dk/), which -is primarily supported by a grant from -[Sygeforsikringen "danmark"](https://www.sygeforsikring.dk/). +[INCEPT (Intensive Care Platform Trial) project](https://incept.dk/), primarily +supported by a grant from [Sygeforsikringen "danmark"](https://www.sygeforsikring.dk/). ## Resources @@ -43,6 +42,17 @@ Clinical Epidemiology describing key methodological considerations in adaptive trials with description of the workflow and a simulation-based example using the package +**Examples:** + +* [Effects of duration of follow-up and lag in data collection on the performance of adaptive clinical trials](https://doi.org/10.1002/pst.2342) - article in Pharmaceutical +Statistics describing a simulation study (with code) using `adaptr` to assess +the performance of adaptive clinical trials according to different +follow-up/data collection lags. +* [Effects of sceptical priors on the performance of adaptive clinical trials with binary outcomes](https://doi.org/10.1002/pst.2387) - +article in Pharmaceutical Statistics describing a simulation study (with code) +using `adaptr` to assess the performance of adaptive clinical trials according to different +sceptical priors. + ## Installation The easiest way is to install from CRAN directly: @@ -53,105 +63,254 @@ install.packages("adaptr") Alternatively, you can install the **development version** from GitHub - this requires the *remotes*-package installed. The development version may contain -additional features not yet available in the CRAN version (including preliminary -functions), but may not be stable or fully documented: +additional features not yet available in the CRAN version, but may not be stable +or fully documented: ```{r eval = FALSE} # install.packages("remotes") remotes::install_github("INCEPTdk/adaptr@dev") ``` -## Example +## Usage and workflow overview -The basic functionality of `adaptr` is illustrated below. +The central functionality of `adaptr` and the typical workflow is illustrated +here. -First, load the library and setup a trial specification using the general -`setup_trial()` function, or one of the special case functions, -`setup_trial_binom()` (used in the example) or `setup_trial_norm()`. +### Setup + +First, the package is loaded and a cluster of parallel workers is initiated by +the `setup_cluster()` function to facilitate parallel computing: ```{r} library(adaptr) +setup_cluster(2) +``` + +### Specify trial design + +Setup a trial specification (defining the trial design and scenario) using +the general `setup_trial()` function, or one of the special case variants using +default priors `setup_trial_binom()` (for binary, binomially distributed +outcomes; used in this example) or `setup_trial_norm()` (for continuous, +normally distributed outcomes). + +```{r} # Setup a trial using a binary, binomially distributed, undesirable outcome binom_trial <- setup_trial_binom( arms = c("Arm A", "Arm B", "Arm C"), - true_ys = c(0.25, 0.20, 0.30), - min_probs = rep(0.15, 3), # Minimum allocation of 15% in all arms + # Scenario with identical outcomes in all arms + true_ys = c(0.25, 0.25, 0.25), + # Response-adaptive randomisation with minimum 20% allocation in all arms + min_probs = rep(0.20, 3), + # Number of patients with data available at each analysis data_looks = seq(from = 300, to = 2000, by = 100), + # Number of patients randomised at each analysis (higher than the numbers + # with data, except at last look, due to follow-up/data collection lag) + randomised_at_looks = c(seq(from = 400, to = 2000, by = 100), 2000), + # Stopping rules for inferiority/superiority not explicitly defined # Stop for equivalence at > 90% probability of differences < 5 %-points equivalence_prob = 0.9, - equivalence_diff = 0.05, - soften_power = 0.5 # Soften allocation ratios + equivalence_diff = 0.05 ) # Print trial specification print(binom_trial, prob_digits = 3) ``` -The resulting trial specification may be calibrated to obtain a specific value -for a certain performance metric (e.g., the Bayesian type 1 error rate for -trial specifications with no between-arm differences) by using the -`calibrate_trial()` function. +### Calibration + +In the example trial specification, there are no true between-arm differences, +and stopping rules for inferiority and superiority are not explicitly defined. +This is intentional, as these stopping rules will be calibrated to obtain a +desired probability of stopping for superiority in the scenario with no +between-arm differences (corresponding to the Bayesian type 1 error rate). Trial +specifications do not necessarily have to be calibrated, and simulations can be +run directly using the `run_trials()` function covered below (or `run_trial()` +for a single simulation). + +Calibration of a trial specification is done using the `calibrate_trial()` +function, which defaults to calibrate constant, symmetrical stopping rules +for inferiority and superiority (expecting a trial specification with +identical outcomes in each arm), but can be used to calibrate any parameter in a +trial specification towards any performance metric. + +```{r} +# Calibrate the trial specification +calibrated_binom_trial <- calibrate_trial( + trial_spec = binom_trial, + n_rep = 1000, # 1000 simulations for each step (more generally recommended) + base_seed = 4131, # Base random seed (for reproducible results) + target = 0.05, # Target value for calibrated metric (default value) + search_range = c(0.9, 1), # Search range for superiority stopping threshold + tol = 0.01, # Tolerance range + dir = -1 # Tolerance range only applies below target +) + +# Print result (to check if calibration is successful) +calibrated_binom_trial +``` + +The calibration is successful - the calibrated, constant stopping threshold for +superiority is printed with the results (`r calibrated_binom_trial$best_x`) and +can be extracted using `calibrated_binom_trial$best_x`. Using the default +calibration functionality, the calibrated, constant stopping threshold for +inferiority is symmetrical, i.e., `1 - stopping threshold for superiority` +(`r 1 - calibrated_binom_trial$best_x`). The calibrated trial specification +may be extracted using `calibrated_binom_trial$best_trial_spec` and, if printed, +will also include the calibrated stopping thresholds. + +Calibration results may be saved (and reloaded) by using the `path` argument, to +avoid unnecessary repeated simulations. + +### Summarising results + +The results of the simulations using the calibrated trial specification +conducted during the calibration procedure may be extracted using +`calibrated_binom_trial$best_sims`. These results can be summarised with several +functions. Most of these functions support different 'selection strategies' for +simulations not ending with superiority, i.e., performance metrics can be +calculated assuming different arms would be used in clinical practice if no arm +is ultimately superior. + +The `check_performance()` function summarises performance metrics in a tidy +`data.frame`, with uncertainty measures (bootstrapped confidence intervals) if +requested. Here, performance metrics are calculated considering the 'best' arm +(i.e., the one with the highest probability of being overall best) selected in +simulations not ending with superiority: + +```{r} +# Calculate performance metrics with uncertainty measures +binom_trial_performance <- check_performance( + calibrated_binom_trial$best_sims, + select_strategy = "best", + uncertainty = TRUE, # Calculate uncertainty measures + n_boot = 1000, # 1000 bootstrap samples (more typically recommended) + ci_width = 0.95, # 95% confidence intervals (default) + boot_seed = "base" # Use same random seed for bootstrapping as for simulations +) + +# Print results +print(binom_trial_performance, digits = 2) +``` +Similar results in `list` format (without uncertainty measures) can be obtained +using the `summary()` method, which comes with a `print()` method providing +formatted results: + +```{r} +binom_trial_summary <- summary( + calibrated_binom_trial$best_sims, + select_strategy = "best" +) + +print(binom_trial_summary) +``` + +Individual simulation results may be extracted in a tidy `data.frame` using +`extract_results()`. + +Finally, the probabilities of different remaining arms and +their statuses (with uncertainty) at the last adaptive analysis can be +summarised using the `check_remaining_arms()` function. + +### Visualising results + +Several visualisation functions are included (all are optional, and all require +the `ggplot2` package installed). -Simulate a single trial using a reproducible random seed: +Convergence and stability of one or more performance metrics may be visually +assessed using `plot_convergence()` function: ```{r} -trial_res <- run_trial(binom_trial, seed = 12345) -print(trial_res, digits = 3) +plot_convergence( + calibrated_binom_trial$best_sims, + metrics = c("size mean", "prob_superior", "prob_equivalence"), + # select_strategy can be specified, but does not affect the chosen metrics +) ``` -Simulate multiple trials using a reproducible random seed: +The empirical cumulative distribution functions for continuous performance +metrics may also be visualised: ```{r} -# Simulate multiple trials - only 10 simulations for speed in the example -trial_res_mult <- run_trials(binom_trial, n_rep = 10, base_seed = 67890) - -# Extract results in a tidy data.frame (1 simulation per row) -# See function documentation for details, including on arm selection in trials -# not ending with a superior arm -extr_res <- extract_results(trial_res_mult) -head(extr_res) - -# Summarise trial results -# See function documentation for details, including on arm selection in trials -# not ending with a superior arm -res_sum <- summary(trial_res_mult) -print(res_sum, digits = 1) +plot_metrics_ecdf( + calibrated_binom_trial$best_sims, + metrics = "size" +) ``` -Performance metrics may also be calculated and returned in a tidy -`data.frame` (with bootstrapped uncertainty measures, if requested) by -the `check_performance()` function, and the empirical cumulative distribution -functions of numerical performance metrics may be plotted by the -`plot_metrics_ecdf()` function. The `plot_convergence()` function may be used to -visually assess stability of performance metrics according to the number of -simulations. Finally, the `check_remaining_arms()` function may be used to -summarise all combinations of remaining arms across multiple simulations. +The status probabilities for the overall trial (or for specific arms) according +to trial progress can be visualised using the `plot_status()` function: + +```{r} +# Overall trial status probabilities +plot_status( + calibrated_binom_trial$best_sims, + x_value = "total n" # Total number of randomised patients at X-axis +) +``` + +Finally, various metrics may be summarised over the progress of one or multiple +trial simulations using the `plot_history()` function, which requires non-sparse +results (the `sparse` argument must be `FALSE` in `calibrate_trials()`, +`run_trials()`, or `run_trial()`, leading to additional results being saved). + +### Use calibrated stopping thresholds in another scenario + +The calibrated stopping thresholds (calibrated in a scenario with no between-arm +differences) may be used to run simulations with the same overall trial +specification, but according to a different scenario (i.e., with between-arm +differences present) to assess performance metrics (including the Bayesian +analogue of power). -Plot trial statuses or history of trial metrics over time: +First, a new trial specification is setup using the same settings as before, +except for between-arm differences and the calibrated stopping thresholds: -```{r plot, fig.align="center", fig.width=6, fig.height=3} -# Simulate multiple trials - 25 simulations only for speed -# sparse = FALSE is required for plot_history (but not plot_status) -trial_res_mult <- run_trials(binom_trial, n_rep = 25, base_seed = 67890, sparse = FALSE) +```{r} +binom_trial_calib_diff <- setup_trial_binom( + arms = c("Arm A", "Arm B", "Arm C"), + true_ys = c(0.25, 0.20, 0.30), # Different outcomes in the arms + min_probs = rep(0.20, 3), + data_looks = seq(from = 300, to = 2000, by = 100), + randomised_at_looks = c(seq(from = 400, to = 2000, by = 100), 2000), + # Stopping rules for inferiority/superiority explicitly defined + # using the calibration results + inferiority = 1 - calibrated_binom_trial$best_x, + superiority = calibrated_binom_trial$best_x, + equivalence_prob = 0.9, + equivalence_diff = 0.05 +) +``` -# Plot overall trial statuses according to the total number -# of patients randomised -plot_status(trial_res_mult, x_value = "total n") +Simulations using the trial specification with calibrated stopping thresholds +and differences present can then be conducted using the `run_trials()` function +and performance metrics calculated as above: -# Plot allocation probabilities at each adaptive look (requires sparse = FALSE) -plot_history(trial_res_mult) +```{r} +binom_trial_diff_sims <- run_trials( + binom_trial_calib_diff, + n_rep = 1000, # 1000 simulations (more generally recommended) + base_seed = 1234 # Reproducible results +) + +check_performance( + binom_trial_diff_sims, + select_strategy = "best", + uncertainty = TRUE, + n_boot = 1000, # 1000 bootstrap samples (more typically recommended) + ci_width = 0.95, + boot_seed = "base" +) ``` -Plotting statuses for individual trial arms and other summary metrics is -possible, too. +Again, simulations may be saved and reloaded using the `path` argument. -Running simulations and extracting and processing results may be done in -parallel by either using the `setup_cluster()` function to set up a cluster of -parallel workers that may be used throughout the session, or with new clusters -each time parallel computation is done by setting the `"mc.cores"` global option -via `options(mc.cores = )` or by using the `cores` argument of many -functions in the package. +Similarly, overall trial statuses for the scenario with differences can be +visualised: + +```{r} +plot_status(binom_trial_diff_sims, x_value = "total n") +``` ## Issues and enhancements @@ -177,7 +336,7 @@ of `adaptr` ## Citation -If using the package, please consider citing it: +If you use the package, please consider citing it: ```{r} citation(package = "adaptr") diff --git a/README.md b/README.md index c1eb272..e9555b7 100644 --- a/README.md +++ b/README.md @@ -19,9 +19,8 @@ clinical trials using adaptive stopping, adaptive arm dropping and/or response-adaptive randomisation. The package has been developed as part of the [INCEPT (Intensive Care -Platform Trial) project](https://incept.dk/), which is primarily -supported by a grant from [Sygeforsikringen -“danmark”](https://www.sygeforsikring.dk/). +Platform Trial) project](https://incept.dk/), primarily supported by a +grant from [Sygeforsikringen “danmark”](https://www.sygeforsikring.dk/). ## Resources @@ -37,6 +36,20 @@ supported by a grant from [Sygeforsikringen considerations in adaptive trials with description of the workflow and a simulation-based example using the package +**Examples:** + +- [Effects of duration of follow-up and lag in data collection on the + performance of adaptive clinical + trials](https://doi.org/10.1002/pst.2342) - article in + Pharmaceutical Statistics describing a simulation study (with code) + using `adaptr` to assess the performance of adaptive clinical trials + according to different follow-up/data collection lags. +- [Effects of sceptical priors on the performance of adaptive clinical + trials with binary outcomes](https://doi.org/10.1002/pst.2387) - + article in Pharmaceutical Statistics describing a simulation study + (with code) using `adaptr` to assess the performance of adaptive + clinical trials according to different sceptical priors. + ## Installation The easiest way is to install from CRAN directly: @@ -47,39 +60,59 @@ install.packages("adaptr") Alternatively, you can install the **development version** from GitHub - this requires the *remotes*-package installed. The development version -may contain additional features not yet available in the CRAN version -(including preliminary functions), but may not be stable or fully -documented: +may contain additional features not yet available in the CRAN version, +but may not be stable or fully documented: ``` r # install.packages("remotes") remotes::install_github("INCEPTdk/adaptr@dev") ``` -## Example +## Usage and workflow overview -The basic functionality of `adaptr` is illustrated below. +The central functionality of `adaptr` and the typical workflow is +illustrated here. -First, load the library and setup a trial specification using the -general `setup_trial()` function, or one of the special case functions, -`setup_trial_binom()` (used in the example) or `setup_trial_norm()`. +### Setup + +First, the package is loaded and a cluster of parallel workers is +initiated by the `setup_cluster()` function to facilitate parallel +computing: ``` r library(adaptr) -#> Loading 'adaptr' package v1.3.2. +#> Loading 'adaptr' package v1.4.0. #> For instructions, type 'help("adaptr")' #> or see https://inceptdk.github.io/adaptr/. +setup_cluster(2) +``` + +### Specify trial design + +Setup a trial specification (defining the trial design and scenario) +using the general `setup_trial()` function, or one of the special case +variants using default priors `setup_trial_binom()` (for binary, +binomially distributed outcomes; used in this example) or +`setup_trial_norm()` (for continuous, normally distributed outcomes). + +``` r # Setup a trial using a binary, binomially distributed, undesirable outcome binom_trial <- setup_trial_binom( arms = c("Arm A", "Arm B", "Arm C"), - true_ys = c(0.25, 0.20, 0.30), - min_probs = rep(0.15, 3), # Minimum allocation of 15% in all arms + # Scenario with identical outcomes in all arms + true_ys = c(0.25, 0.25, 0.25), + # Response-adaptive randomisation with minimum 20% allocation in all arms + min_probs = rep(0.20, 3), + # Number of patients with data available at each analysis data_looks = seq(from = 300, to = 2000, by = 100), + # Number of patients randomised at each analysis (higher than the numbers + # with data, except at last look, due to follow-up/data collection lag) + randomised_at_looks = c(seq(from = 400, to = 2000, by = 100), 2000), + # Stopping rules for inferiority/superiority not explicitly defined # Stop for equivalence at > 90% probability of differences < 5 %-points equivalence_prob = 0.9, - equivalence_diff = 0.05, - soften_power = 0.5 # Soften allocation ratios + equivalence_diff = 0.05 ) # Print trial specification @@ -87,176 +120,364 @@ print(binom_trial, prob_digits = 3) #> Trial specification: generic binomially distributed outcome trial #> * Undesirable outcome #> * No common control arm -#> * Best arm: Arm B +#> * Best arms: Arm A and Arm B and Arm C #> #> Arms, true outcomes, starting allocation probabilities #> and allocation probability limits: #> arms true_ys start_probs fixed_probs min_probs max_probs -#> Arm A 0.25 0.333 NA 0.15 NA -#> Arm B 0.20 0.333 NA 0.15 NA -#> Arm C 0.30 0.333 NA 0.15 NA +#> Arm A 0.25 0.333 NA 0.2 NA +#> Arm B 0.25 0.333 NA 0.2 NA +#> Arm C 0.25 0.333 NA 0.2 NA #> #> Maximum sample size: 2000 #> Maximum number of data looks: 18 #> Planned data looks after: 300, 400, 500, 600, 700, 800, 900, 1000, 1100, 1200, 1300, 1400, 1500, 1600, 1700, 1800, 1900, 2000 patients have reached follow-up -#> Number of patients randomised at each look: 300, 400, 500, 600, 700, 800, 900, 1000, 1100, 1200, 1300, 1400, 1500, 1600, 1700, 1800, 1900, 2000 +#> Number of patients randomised at each look: 400, 500, 600, 700, 800, 900, 1000, 1100, 1200, 1300, 1400, 1500, 1600, 1700, 1800, 1900, 2000, 2000 #> #> Superiority threshold: 0.99 (all analyses) #> Inferiority threshold: 0.01 (all analyses) #> Equivalence threshold: 0.9 (all analyses) (no common control) #> Absolute equivalence difference: 0.05 #> No futility threshold (not relevant - no common control) -#> Soften power for all analyses: 0.5 +#> Soften power for all analyses: 1 (no softening) ``` -The resulting trial specification may be calibrated to obtain a specific -value for a certain performance metric (e.g., the Bayesian type 1 error -rate for trial specifications with no between-arm differences) by using -the `calibrate_trial()` function. - -Simulate a single trial using a reproducible random seed: +### Calibration + +In the example trial specification, there are no true between-arm +differences, and stopping rules for inferiority and superiority are not +explicitly defined. This is intentional, as these stopping rules will be +calibrated to obtain a desired probability of stopping for superiority +in the scenario with no between-arm differences (corresponding to the +Bayesian type 1 error rate). Trial specifications do not necessarily +have to be calibrated, and simulations can be run directly using the +`run_trials()` function covered below (or `run_trial()` for a single +simulation). + +Calibration of a trial specification is done using the +`calibrate_trial()` function, which defaults to calibrate constant, +symmetrical stopping rules for inferiority and superiority (expecting a +trial specification with identical outcomes in each arm), but can be +used to calibrate any parameter in a trial specification towards any +performance metric. ``` r -trial_res <- run_trial(binom_trial, seed = 12345) -print(trial_res, digits = 3) -#> Single simulation result: generic binomially distributed outcome trial -#> * Undesirable outcome -#> * No common control arm -#> -#> Final status: inconclusive, stopped at final allowed adaptive analysis -#> Final/maximum allowed sample sizes: 2000/2000 (100.0%) -#> Available outcome data at last adaptive analysis: 2000/2000 (100.0%) -#> -#> Trial results overview: -#> arms true_ys final_status status_look status_probs final_alloc -#> Arm A 0.25 active NA NA 0.194 -#> Arm B 0.20 active NA NA 0.656 -#> Arm C 0.30 inferior 2000 0.007 0.150 +# Calibrate the trial specification +calibrated_binom_trial <- calibrate_trial( + trial_spec = binom_trial, + n_rep = 1000, # 1000 simulations for each step (more generally recommended) + base_seed = 4131, # Base random seed (for reproducible results) + target = 0.05, # Target value for calibrated metric (default value) + search_range = c(0.9, 1), # Search range for superiority stopping threshold + tol = 0.01, # Tolerance range + dir = -1 # Tolerance range only applies below target +) + +# Print result (to check if calibration is successful) +calibrated_binom_trial +#> Trial calibration: +#> * Result: calibration successful +#> * Best x: 0.9814318 +#> * Best y: 0.048 #> -#> Esimates from final analysis (all patients): -#> arms sum_ys_all ns_all raw_ests_all post_ests_all post_errs_all lo_cri_all -#> Arm A 180 742 0.243 0.243 0.0161 0.213 -#> Arm B 178 841 0.212 0.212 0.0141 0.185 -#> Arm C 113 417 0.271 0.272 0.0221 0.230 -#> hi_cri_all -#> 0.274 -#> 0.240 -#> 0.316 +#> Central settings: +#> * Target: 0.05 +#> * Tolerance: 0.01 (at or below target, range: 0.04 to 0.05) +#> * Search range: 0.9 to 1 +#> * Gaussian process controls: +#> * - resolution: 5000 +#> * - kappa: 0.5 +#> * - pow: 1.95 +#> * - lengthscale: 1 (constant) +#> * - x scaled: yes +#> * Noisy: no +#> * Narrowing: yes #> -#> Estimates from last adaptive analysis including each arm: -#> arms sum_ys ns raw_ests post_ests post_errs lo_cri hi_cri -#> Arm A 180 742 0.243 0.243 0.0159 0.213 0.275 -#> Arm B 178 841 0.212 0.212 0.0141 0.185 0.241 -#> Arm C 113 417 0.271 0.271 0.0215 0.230 0.316 +#> Calibration/simulation details: +#> * Total evaluations: 7 (previous + grid + iterations) +#> * Repetitions: 1000 +#> * Calibration time: 3.66 mins +#> * Base random seed: 4131 #> -#> Simulation details: -#> * Random seed: 12345 -#> * Credible interval width: 95% -#> * Number of posterior draws: 5000 -#> * Posterior estimation method: medians with MAD-SDs +#> See 'help("calibrate_trial")' for details. ``` -Simulate multiple trials using a reproducible random seed: +The calibration is successful - the calibrated, constant stopping +threshold for superiority is printed with the results (0.9814318) and +can be extracted using `calibrated_binom_trial$best_x`. Using the +default calibration functionality, the calibrated, constant stopping +threshold for inferiority is symmetrical, i.e., +`1 - stopping threshold for superiority` (0.0185682). The calibrated +trial specification may be extracted using +`calibrated_binom_trial$best_trial_spec` and, if printed, will also +include the calibrated stopping thresholds. + +Calibration results may be saved (and reloaded) by using the `path` +argument, to avoid unnecessary repeated simulations. + +### Summarising results + +The results of the simulations using the calibrated trial specification +conducted during the calibration procedure may be extracted using +`calibrated_binom_trial$best_sims`. These results can be summarised with +several functions. Most of these functions support different ‘selection +strategies’ for simulations not ending with superiority, i.e., +performance metrics can be calculated assuming different arms would be +used in clinical practice if no arm is ultimately superior. + +The `check_performance()` function summarises performance metrics in a +tidy `data.frame`, with uncertainty measures (bootstrapped confidence +intervals) if requested. Here, performance metrics are calculated +considering the ‘best’ arm (i.e., the one with the highest probability +of being overall best) selected in simulations not ending with +superiority: ``` r -# Simulate multiple trials - only 10 simulations for speed in the example -trial_res_mult <- run_trials(binom_trial, n_rep = 10, base_seed = 67890) - -# Extract results in a tidy data.frame (1 simulation per row) -# See function documentation for details, including on arm selection in trials -# not ending with a superior arm -extr_res <- extract_results(trial_res_mult) -head(extr_res) -#> sim final_n sum_ys ratio_ys final_status superior_arm selected_arm -#> 1 1 1400 332 0.2371429 superiority Arm B Arm B -#> 2 2 900 196 0.2177778 equivalence -#> 3 3 2000 441 0.2205000 max -#> 4 4 1300 274 0.2107692 superiority Arm B Arm B -#> 5 5 1400 354 0.2528571 equivalence -#> 6 6 1700 373 0.2194118 superiority Arm B Arm B -#> sq_err sq_err_te -#> 1 8.118136e-06 NA -#> 2 NA NA -#> 3 NA NA -#> 4 3.801078e-04 NA -#> 5 NA NA -#> 6 3.871855e-05 NA - -# Summarise trial results -# See function documentation for details, including on arm selection in trials -# not ending with a superior arm -res_sum <- summary(trial_res_mult) -print(res_sum, digits = 1) +# Calculate performance metrics with uncertainty measures +binom_trial_performance <- check_performance( + calibrated_binom_trial$best_sims, + select_strategy = "best", + uncertainty = TRUE, # Calculate uncertainty measures + n_boot = 1000, # 1000 bootstrap samples (more typically recommended) + ci_width = 0.95, # 95% confidence intervals (default) + boot_seed = "base" # Use same random seed for bootstrapping as for simulations +) + +# Print results +print(binom_trial_performance, digits = 2) +#> metric est err_sd err_mad lo_ci hi_ci +#> 1 n_summarised 1000.00 0.00 0.00 1000.00 1000.00 +#> 2 size_mean 1749.60 11.36 10.97 1727.20 1772.10 +#> 3 size_sd 373.74 9.64 9.74 355.15 392.58 +#> 4 size_median 2000.00 0.00 0.00 2000.00 2000.00 +#> 5 size_p25 1400.00 52.43 0.00 1400.00 1500.00 +#> 6 size_p75 2000.00 0.00 0.00 2000.00 2000.00 +#> 7 size_p0 400.00 NA NA NA NA +#> 8 size_p100 2000.00 NA NA NA NA +#> 9 sum_ys_mean 438.69 2.95 2.85 432.74 444.66 +#> 10 sum_ys_sd 96.20 2.42 2.37 91.28 100.79 +#> 11 sum_ys_median 486.00 1.98 2.97 483.00 490.00 +#> 12 sum_ys_p25 364.75 10.95 9.64 352.00 395.00 +#> 13 sum_ys_p75 506.00 1.15 1.48 504.00 508.00 +#> 14 sum_ys_p0 88.00 NA NA NA NA +#> 15 sum_ys_p100 565.00 NA NA NA NA +#> 16 ratio_ys_mean 0.25 0.00 0.00 0.25 0.25 +#> 17 ratio_ys_sd 0.01 0.00 0.00 0.01 0.01 +#> 18 ratio_ys_median 0.25 0.00 0.00 0.25 0.25 +#> 19 ratio_ys_p25 0.24 0.00 0.00 0.24 0.24 +#> 20 ratio_ys_p75 0.26 0.00 0.00 0.26 0.26 +#> 21 ratio_ys_p0 0.20 NA NA NA NA +#> 22 ratio_ys_p100 0.30 NA NA NA NA +#> 23 prob_conclusive 0.43 0.02 0.01 0.40 0.46 +#> 24 prob_superior 0.05 0.01 0.01 0.04 0.06 +#> 25 prob_equivalence 0.38 0.02 0.01 0.35 0.41 +#> 26 prob_futility 0.00 0.00 0.00 0.00 0.00 +#> 27 prob_max 0.57 0.02 0.01 0.54 0.60 +#> 28 prob_select_arm_Arm A 0.32 0.02 0.01 0.29 0.35 +#> 29 prob_select_arm_Arm B 0.31 0.01 0.01 0.28 0.34 +#> 30 prob_select_arm_Arm C 0.37 0.02 0.02 0.34 0.40 +#> 31 prob_select_none 0.00 0.00 0.00 0.00 0.00 +#> 32 rmse 0.02 0.00 0.00 0.02 0.02 +#> 33 rmse_te NA NA NA NA NA +#> 34 mae 0.01 0.00 0.00 0.01 0.01 +#> 35 mae_te NA NA NA NA NA +#> 36 idp NA NA NA NA NA +``` + +Similar results in `list` format (without uncertainty measures) can be +obtained using the `summary()` method, which comes with a `print()` +method providing formatted results: + +``` r +binom_trial_summary <- summary( + calibrated_binom_trial$best_sims, + select_strategy = "best" +) + +print(binom_trial_summary) #> Multiple simulation results: generic binomially distributed outcome trial #> * Undesirable outcome -#> * Number of simulations: 10 -#> * Number of simulations summarised: 10 (all trials) +#> * Number of simulations: 1000 +#> * Number of simulations summarised: 1000 (all trials) #> * No common control arm -#> * Selection strategy: no selection if no superior arm +#> * Selection strategy: best remaining available #> * Treatment effect compared to: no comparison #> -#> Performance metrics (using posterior estimates from last adaptive analysis): -#> * Sample sizes: mean 1350.0 (SD: 445.3) | median 1350.0 (IQR: 1025.0 to 1625.0) [range: 700.0 to 2000.0] -#> * Total summarised outcomes: mean 305.3 (SD: 103.8) | median 303.0 (IQR: 231.0 to 368.2) [range: 156.0 to 463.0] -#> * Total summarised outcome rates: mean 0.226 (SD: 0.013) | median 0.222 (IQR: 0.218 to 0.231) [range: 0.211 to 0.253] -#> * Conclusive: 80.0% -#> * Superiority: 40.0% -#> * Equivalence: 40.0% +#> Performance metrics (using posterior estimates from final analysis [all patients]): +#> * Sample sizes: mean 1749.6 (SD: 373.7) | median 2000.0 (IQR: 1400.0 to 2000.0) [range: 400.0 to 2000.0] +#> * Total summarised outcomes: mean 438.7 (SD: 96.2) | median 486.0 (IQR: 364.8 to 506.0) [range: 88.0 to 565.0] +#> * Total summarised outcome rates: mean 0.251 (SD: 0.011) | median 0.250 (IQR: 0.244 to 0.258) [range: 0.198 to 0.295] +#> * Conclusive: 42.9% +#> * Superiority: 4.8% +#> * Equivalence: 38.1% #> * Futility: 0.0% [not assessed] -#> * Inconclusive at max sample size: 20.0% -#> * Selection probabilities: Arm A: 0.0% | Arm B: 40.0% | Arm C: 0.0% | None: 60.0% -#> * RMSE: 0.01675 -#> * RMSE treatment effect: not estimated -#> * Ideal design percentage: 100.0% +#> * Inconclusive at max sample size: 57.1% +#> * Selection probabilities: Arm A: 31.8% | Arm B: 31.0% | Arm C: 37.2% | None: 0.0% +#> * RMSE / MAE: 0.01730 / 0.01102 +#> * RMSE / MAE treatment effect: not estimated / not estimated +#> * Ideal design percentage: not estimable #> #> Simulation details: -#> * Simulation time: 0.475 secs -#> * Base random seed: 67890 +#> * Simulation time: 33.1 secs +#> * Base random seed: 4131 #> * Credible interval width: 95% #> * Number of posterior draws: 5000 #> * Estimation method: posterior medians with MAD-SDs ``` -Performance metrics may also be calculated and returned in a tidy -`data.frame` (with bootstrapped uncertainty measures, if requested) by -the `check_performance()` function, and the empirical cumulative -distribution functions of numerical performance metrics may be plotted -by the `plot_metrics_ecdf()` function. The `plot_convergence()` function -may be used to visually assess stability of performance metrics -according to the number of simulations. Finally, the -`check_remaining_arms()` function may be used to summarise all -combinations of remaining arms across multiple simulations. +Individual simulation results may be extracted in a tidy `data.frame` +using `extract_results()`. + +Finally, the probabilities of different remaining arms and their +statuses (with uncertainty) at the last adaptive analysis can be +summarised using the `check_remaining_arms()` function. -Plot trial statuses or history of trial metrics over time: +### Visualising results + +Several visualisation functions are included (all are optional, and all +require the `ggplot2` package installed). + +Convergence and stability of one or more performance metrics may be +visually assessed using `plot_convergence()` function: ``` r -# Simulate multiple trials - 25 simulations only for speed -# sparse = FALSE is required for plot_history (but not plot_status) -trial_res_mult <- run_trials(binom_trial, n_rep = 25, base_seed = 67890, sparse = FALSE) +plot_convergence( + calibrated_binom_trial$best_sims, + metrics = c("size mean", "prob_superior", "prob_equivalence"), + # select_strategy can be specified, but does not affect the chosen metrics +) +``` + + -# Plot overall trial statuses according to the total number -# of patients randomised -plot_status(trial_res_mult, x_value = "total n") +The empirical cumulative distribution functions for continuous +performance metrics may also be visualised: + +``` r +plot_metrics_ecdf( + calibrated_binom_trial$best_sims, + metrics = "size" +) ``` - + + +The status probabilities for the overall trial (or for specific arms) +according to trial progress can be visualised using the `plot_status()` +function: ``` r -# Plot allocation probabilities at each adaptive look (requires sparse = FALSE) -plot_history(trial_res_mult) +# Overall trial status probabilities +plot_status( + calibrated_binom_trial$best_sims, + x_value = "total n" # Total number of randomised patients at X-axis +) ``` - + + +Finally, various metrics may be summarised over the progress of one or +multiple trial simulations using the `plot_history()` function, which +requires non-sparse results (the `sparse` argument must be `FALSE` in +`calibrate_trials()`, `run_trials()`, or `run_trial()`, leading to +additional results being saved). -Plotting statuses for individual trial arms and other summary metrics is -possible, too. +### Use calibrated stopping thresholds in another scenario + +The calibrated stopping thresholds (calibrated in a scenario with no +between-arm differences) may be used to run simulations with the same +overall trial specification, but according to a different scenario +(i.e., with between-arm differences present) to assess performance +metrics (including the Bayesian analogue of power). + +First, a new trial specification is setup using the same settings as +before, except for between-arm differences and the calibrated stopping +thresholds: + +``` r +binom_trial_calib_diff <- setup_trial_binom( + arms = c("Arm A", "Arm B", "Arm C"), + true_ys = c(0.25, 0.20, 0.30), # Different outcomes in the arms + min_probs = rep(0.20, 3), + data_looks = seq(from = 300, to = 2000, by = 100), + randomised_at_looks = c(seq(from = 400, to = 2000, by = 100), 2000), + # Stopping rules for inferiority/superiority explicitly defined + # using the calibration results + inferiority = 1 - calibrated_binom_trial$best_x, + superiority = calibrated_binom_trial$best_x, + equivalence_prob = 0.9, + equivalence_diff = 0.05 +) +``` + +Simulations using the trial specification with calibrated stopping +thresholds and differences present can then be conducted using the +`run_trials()` function and performance metrics calculated as above: + +``` r +binom_trial_diff_sims <- run_trials( + binom_trial_calib_diff, + n_rep = 1000, # 1000 simulations (more generally recommended) + base_seed = 1234 # Reproducible results +) + +check_performance( + binom_trial_diff_sims, + select_strategy = "best", + uncertainty = TRUE, + n_boot = 1000, # 1000 bootstrap samples (more typically recommended) + ci_width = 0.95, + boot_seed = "base" +) +#> metric est err_sd err_mad lo_ci hi_ci +#> 1 n_summarised 1000.000 0.000 0.000 1000.000 1000.000 +#> 2 size_mean 1242.300 16.620 16.976 1209.895 1273.025 +#> 3 size_sd 531.190 7.251 7.604 516.617 544.091 +#> 4 size_median 1200.000 22.220 0.000 1200.000 1300.000 +#> 5 size_p25 800.000 36.095 0.000 700.000 800.000 +#> 6 size_p75 1700.000 42.453 0.000 1700.000 1800.000 +#> 7 size_p0 400.000 NA NA NA NA +#> 8 size_p100 2000.000 NA NA NA NA +#> 9 sum_ys_mean 284.999 3.695 3.726 277.724 291.991 +#> 10 sum_ys_sd 117.265 1.701 1.732 113.765 120.311 +#> 11 sum_ys_median 279.000 5.268 4.448 269.500 289.512 +#> 12 sum_ys_p25 186.000 6.682 7.413 174.000 197.019 +#> 13 sum_ys_p75 390.000 7.633 7.413 374.000 402.250 +#> 14 sum_ys_p0 81.000 NA NA NA NA +#> 15 sum_ys_p100 519.000 NA NA NA NA +#> 16 ratio_ys_mean 0.232 0.000 0.001 0.231 0.233 +#> 17 ratio_ys_sd 0.016 0.000 0.000 0.015 0.017 +#> 18 ratio_ys_median 0.230 0.001 0.000 0.230 0.232 +#> 19 ratio_ys_p25 0.221 0.000 0.000 0.220 0.222 +#> 20 ratio_ys_p75 0.242 0.001 0.001 0.240 0.243 +#> 21 ratio_ys_p0 0.195 NA NA NA NA +#> 22 ratio_ys_p100 0.298 NA NA NA NA +#> 23 prob_conclusive 0.877 0.011 0.010 0.857 0.898 +#> 24 prob_superior 0.731 0.014 0.015 0.706 0.759 +#> 25 prob_equivalence 0.146 0.011 0.011 0.125 0.167 +#> 26 prob_futility 0.000 0.000 0.000 0.000 0.000 +#> 27 prob_max 0.123 0.011 0.010 0.102 0.143 +#> 28 prob_select_arm_Arm A 0.038 0.006 0.006 0.026 0.049 +#> 29 prob_select_arm_Arm B 0.962 0.006 0.006 0.951 0.974 +#> 30 prob_select_arm_Arm C 0.000 0.000 0.000 0.000 0.000 +#> 31 prob_select_none 0.000 0.000 0.000 0.000 0.000 +#> 32 rmse 0.020 0.001 0.001 0.019 0.022 +#> 33 rmse_te NA NA NA NA NA +#> 34 mae 0.011 0.000 0.000 0.010 0.012 +#> 35 mae_te NA NA NA NA NA +#> 36 idp 98.100 0.306 0.297 97.549 98.700 +``` + +Again, simulations may be saved and reloaded using the `path` argument. + +Similarly, overall trial statuses for the scenario with differences can +be visualised: + +``` r +plot_status(binom_trial_diff_sims, x_value = "total n") +``` -Running simulations and extracting and processing results may be done in -parallel by either using the `setup_cluster()` function to set up a -cluster of parallel workers that may be used throughout the session, or -with new clusters each time parallel computation is done by setting the -`"mc.cores"` global option via `options(mc.cores = )` or by -using the `cores` argument of many functions in the package. + ## Issues and enhancements @@ -288,7 +509,7 @@ Changes to the code base should follow these steps: ## Citation -If using the package, please consider citing it: +If you use the package, please consider citing it: ``` r citation(package = "adaptr") diff --git a/_pkgdown.yml b/_pkgdown.yml index 6d602d3..17c6a12 100644 --- a/_pkgdown.yml +++ b/_pkgdown.yml @@ -15,6 +15,10 @@ reference: contents: - 'adaptr-package' +- title: Parallelisation + contents: + - setup_cluster + - title: Setup, calibrate, and run trials contents: - starts_with("setup_trial") @@ -42,6 +46,6 @@ reference: - title: Helper functions contents: - - setup_cluster - update_saved_trials + - update_saved_calibration - find_beta_params diff --git a/cran-comments.md b/cran-comments.md index 48a437b..47ae772 100644 --- a/cran-comments.md +++ b/cran-comments.md @@ -1,6 +1,7 @@ ## Release summary -This is a patch release with bug fixes and documentation updates. +This is a minor release implementing new functionality, and including bug fixes, +updates to documentation, argument checking and test coverage. ## R CMD check results diff --git a/inst/testdata/binom___calibration___setup2_arms__no_difference___rar.RData b/inst/testdata/binom___calibration___setup2_arms__no_difference___rar.RData new file mode 100644 index 0000000..9e1c53a Binary files /dev/null and b/inst/testdata/binom___calibration___setup2_arms__no_difference___rar.RData differ diff --git a/inst/testdata/binom__results__3_arms__common_control__equivalence__futility__softened.RData b/inst/testdata/binom__results__3_arms__common_control__equivalence__futility__softened.RData index 6046a8b..6e33aad 100644 Binary files a/inst/testdata/binom__results__3_arms__common_control__equivalence__futility__softened.RData and b/inst/testdata/binom__results__3_arms__common_control__equivalence__futility__softened.RData differ diff --git a/inst/testdata/binom__results__3_arms__no_control__equivalence__softened.RData b/inst/testdata/binom__results__3_arms__no_control__equivalence__softened.RData index 57be954..1b5ce4e 100644 Binary files a/inst/testdata/binom__results__3_arms__no_control__equivalence__softened.RData and b/inst/testdata/binom__results__3_arms__no_control__equivalence__softened.RData differ diff --git a/inst/testdata/binom__results__3_arms__no_control__equivalence__softened__sparse.RData b/inst/testdata/binom__results__3_arms__no_control__equivalence__softened__sparse.RData index 9756948..aa6b564 100644 Binary files a/inst/testdata/binom__results__3_arms__no_control__equivalence__softened__sparse.RData and b/inst/testdata/binom__results__3_arms__no_control__equivalence__softened__sparse.RData differ diff --git a/inst/testdata/binom__results__3_arms__no_control__equivalence__stopping.RData b/inst/testdata/binom__results__3_arms__no_control__equivalence__stopping.RData new file mode 100644 index 0000000..ed26ce2 Binary files /dev/null and b/inst/testdata/binom__results__3_arms__no_control__equivalence__stopping.RData differ diff --git a/inst/testdata/binom__setup__3_arms__common_control__equivalence__futility__softened.RData b/inst/testdata/binom__setup__3_arms__common_control__equivalence__futility__softened.RData index f51195e..7ac840c 100644 Binary files a/inst/testdata/binom__setup__3_arms__common_control__equivalence__futility__softened.RData and b/inst/testdata/binom__setup__3_arms__common_control__equivalence__futility__softened.RData differ diff --git a/inst/testdata/binom__setup__3_arms__no_control__equivalence__softened.RData b/inst/testdata/binom__setup__3_arms__no_control__equivalence__softened.RData index fca8479..84f25eb 100644 Binary files a/inst/testdata/binom__setup__3_arms__no_control__equivalence__softened.RData and b/inst/testdata/binom__setup__3_arms__no_control__equivalence__softened.RData differ diff --git a/inst/testdata/norm__results__3_arms__common_control__fixed__all_arms_fixed.RData b/inst/testdata/norm__results__3_arms__common_control__fixed__all_arms_fixed.RData index fbff65c..ef6dfe8 100644 Binary files a/inst/testdata/norm__results__3_arms__common_control__fixed__all_arms_fixed.RData and b/inst/testdata/norm__results__3_arms__common_control__fixed__all_arms_fixed.RData differ diff --git a/inst/testdata/norm__setup__3_arms__common_control__fixed__all_arms_fixed.RData b/inst/testdata/norm__setup__3_arms__common_control__fixed__all_arms_fixed.RData index 6d06e29..a0573b1 100644 Binary files a/inst/testdata/norm__setup__3_arms__common_control__fixed__all_arms_fixed.RData and b/inst/testdata/norm__setup__3_arms__common_control__fixed__all_arms_fixed.RData differ diff --git a/inst/testdata/norm__setup__3_arms__common_control__matched__varying_probs.RData b/inst/testdata/norm__setup__3_arms__common_control__matched__varying_probs.RData index c61fd53..dcaf0f2 100644 Binary files a/inst/testdata/norm__setup__3_arms__common_control__matched__varying_probs.RData and b/inst/testdata/norm__setup__3_arms__common_control__matched__varying_probs.RData differ diff --git a/man/adaptr-package.Rd b/man/adaptr-package.Rd index ff3802a..f445fa9 100644 --- a/man/adaptr-package.Rd +++ b/man/adaptr-package.Rd @@ -2,8 +2,8 @@ % Please edit documentation in R/adaptr-package.R \docType{package} \name{adaptr-package} -\alias{adaptr-package} \alias{adaptr} +\alias{adaptr-package} \title{adaptr: Adaptive Trial Simulator} \description{ \if{html}{ @@ -76,6 +76,16 @@ trials. J Clin Epidemiol. \doi{10.1016/j.jclinepi.2022.11.002} \href{https://inceptdk.github.io/adaptr/}{Website/manual} \href{https://github.com/INCEPTdk/adaptr/}{GitHub repository} + +\strong{Examples of studies using \code{adaptr}:} + +Granholm A, Lange T, Harhay MO, Jensen AKG, Perner A, Møller MH, Kaas-Hansen +BS (2023). Effects of duration of follow-up and lag in data collection on the +performance of adaptive clinical trials. Pharm Stat. \doi{10.1002/pst.2342} + +Granholm A, Lange T, Harhay MO, Perner A, Møller MH, Kaas-Hansen BS (2024). +Effects of sceptical priors on the performance of adaptive clinical trials +with binary outcomes. Pharm Stat. \doi{10.1002/pst.2387} } \seealso{ \code{\link[=setup_cluster]{setup_cluster()}}, \code{\link[=setup_trial]{setup_trial()}}, \code{\link[=setup_trial_binom]{setup_trial_binom()}}, @@ -84,3 +94,18 @@ trials. J Clin Epidemiol. \doi{10.1016/j.jclinepi.2022.11.002} \code{\link[=check_remaining_arms]{check_remaining_arms()}}, \code{\link[=plot_convergence]{plot_convergence()}}, \code{\link[=plot_metrics_ecdf]{plot_metrics_ecdf()}}, \code{\link[=print]{print()}}, \code{\link[=plot_status]{plot_status()}}, \code{\link[=plot_history]{plot_history()}}. } +\author{ +\strong{Maintainer}: Anders Granholm \email{andersgran@gmail.com} (\href{https://orcid.org/0000-0001-5799-7655}{ORCID}) + +Authors: +\itemize{ + \item Benjamin Skov Kaas-Hansen \email{epiben@hey.com} (\href{https://orcid.org/0000-0003-1023-0371}{ORCID}) +} + +Other contributors: +\itemize{ + \item Aksel Karl Georg Jensen \email{akje@sund.ku.dk} (\href{https://orcid.org/0000-0002-1459-0465}{ORCID}) [contributor] + \item Theis Lange \email{thlan@sund.ku.dk} (\href{https://orcid.org/0000-0001-6807-8347}{ORCID}) [contributor] +} + +} diff --git a/man/check_performance.Rd b/man/check_performance.Rd index 806ef56..d3e8b33 100644 --- a/man/check_performance.Rd +++ b/man/check_performance.Rd @@ -74,19 +74,19 @@ available in the trial.} \item{te_comp}{character string, treatment-effect comparator. Can be either \code{NULL} (the default) in which case the \strong{first} \code{control} arm is used for trial designs with a common control arm, or a string naming a single trial -\code{arm}. Will be used when calculating \code{sq_err_te} (the squared error of the -treatment effect comparing the selected arm to the comparator arm, as -described below).} +\code{arm}. Will be used when calculating \code{err_te} and \code{sq_err_te} (the error +and the squared error of the treatment effect comparing the selected arm to +the comparator arm, as described below).} \item{raw_ests}{single logical. If \code{FALSE} (default), the posterior estimates (\code{post_ests} or \code{post_ests_all}, see \code{\link[=setup_trial]{setup_trial()}} -and \code{\link[=run_trial]{run_trial()}}) will be used to calculate \code{sq_err} (the squared error of -the estimated compared to the specified effect in the selected arm) and -\code{sq_err_te} (the squared error of the treatment effect comparing the -selected arm to the comparator arm, as described for \code{te_comp} and below). -If \code{TRUE}, the raw estimates (\code{raw_ests} or \code{raw_ests_all}, see -\code{\link[=setup_trial]{setup_trial()}} and \code{\link[=run_trial]{run_trial()}}) will be used instead of the posterior -estimates.} +and \code{\link[=run_trial]{run_trial()}}) will be used to calculate \code{err} and \code{sq_err} (the error +and the squared error of the estimated compared to the specified effect in +the selected arm) and \code{err_te} and \code{sq_err_te} (the error and the squared +error of the treatment effect comparing the selected arm to the comparator +arm, as described for \code{te_comp} and below). If \code{TRUE}, the raw estimates +(\code{raw_ests} or \code{raw_ests_all}, see \code{\link[=setup_trial]{setup_trial()}} and \code{\link[=run_trial]{run_trial()}}) will +be used instead of the posterior estimates.} \item{final_ests}{single logical. If \code{TRUE} (recommended) the final estimates calculated using outcome data from all patients randomised when trials are @@ -152,9 +152,9 @@ following four columns if \code{uncertainty = TRUE}: \code{"err_sd"}(bootstrappe SDs), \code{"err_mad"} (bootstrapped MAD-SDs, as described in \code{\link[=setup_trial]{setup_trial()}} and \code{\link[stats:mad]{stats::mad()}}), \code{"lo_ci"}, and \code{"hi_ci"}, the latter two corresponding to the lower/upper limits of the percentile-based bootstrapped confidence -intervals. Bootstrap estimates are \strong{not} calculated for the mininum +intervals. Bootstrap estimates are \strong{not} calculated for the minimum (\verb{_p0}) and maximum values (\verb{_p100}) of \code{size}, \code{sum_ys}, and \code{ratio_ys}, -as non-parametric bootstrapping for mininum/maximum values is not +as non-parametric bootstrapping for minimum/maximum values is not sensible - bootstrap estimates for these values will be \code{NA}.\cr The following performance metrics are calculated: \itemize{ @@ -193,7 +193,10 @@ results are \code{restricted}. selection, according to the specified selection strategy. Contains one element per \code{arm}, named \verb{prob_select_arm_} and \code{prob_select_none} for the probability of selecting no arm. -\item \code{rmse}, \code{rmse_te}: the root mean squared error of the estimates for +\item \code{rmse}, \code{rmse_te}: the root mean squared errors of the estimates for +the selected arm and for the treatment effect, as described in +\code{\link[=extract_results]{extract_results()}}. +\item \code{mae}, \code{mae_te}: the median absolute errors of the estimates for the selected arm and for the treatment effect, as described in \code{\link[=extract_results]{extract_results()}}. \item \code{idp}: the ideal design percentage (IDP; 0-100\%), see \strong{Details}. diff --git a/man/check_remaining_arms.Rd b/man/check_remaining_arms.Rd index 506511b..2f7a7fe 100644 --- a/man/check_remaining_arms.Rd +++ b/man/check_remaining_arms.Rd @@ -40,7 +40,7 @@ any analysis, and arms dropped for equivalence at earlier analyses in trials with a common \code{control}) across multiple simulated trial results. The function supplements the \code{\link[=extract_results]{extract_results()}}, \code{\link[=check_performance]{check_performance()}}, and \code{\link[=summary]{summary()}} functions, and is especially useful for designs with \verb{> 2} arms, -where it provides details that the other functionality mentioned do not. +where it provides details that the other functions mentioned do not. } \examples{ # Setup a trial specification diff --git a/man/extract_results.Rd b/man/extract_results.Rd index 4510b3f..d6f03db 100644 --- a/man/extract_results.Rd +++ b/man/extract_results.Rd @@ -69,19 +69,19 @@ available in the trial.} \item{te_comp}{character string, treatment-effect comparator. Can be either \code{NULL} (the default) in which case the \strong{first} \code{control} arm is used for trial designs with a common control arm, or a string naming a single trial -\code{arm}. Will be used when calculating \code{sq_err_te} (the squared error of the -treatment effect comparing the selected arm to the comparator arm, as -described below).} +\code{arm}. Will be used when calculating \code{err_te} and \code{sq_err_te} (the error +and the squared error of the treatment effect comparing the selected arm to +the comparator arm, as described below).} \item{raw_ests}{single logical. If \code{FALSE} (default), the posterior estimates (\code{post_ests} or \code{post_ests_all}, see \code{\link[=setup_trial]{setup_trial()}} -and \code{\link[=run_trial]{run_trial()}}) will be used to calculate \code{sq_err} (the squared error of -the estimated compared to the specified effect in the selected arm) and -\code{sq_err_te} (the squared error of the treatment effect comparing the -selected arm to the comparator arm, as described for \code{te_comp} and below). -If \code{TRUE}, the raw estimates (\code{raw_ests} or \code{raw_ests_all}, see -\code{\link[=setup_trial]{setup_trial()}} and \code{\link[=run_trial]{run_trial()}}) will be used instead of the posterior -estimates.} +and \code{\link[=run_trial]{run_trial()}}) will be used to calculate \code{err} and \code{sq_err} (the error +and the squared error of the estimated compared to the specified effect in +the selected arm) and \code{err_te} and \code{sq_err_te} (the error and the squared +error of the treatment effect comparing the selected arm to the comparator +arm, as described for \code{te_comp} and below). If \code{TRUE}, the raw estimates +(\code{raw_ests} or \code{raw_ests_all}, see \code{\link[=setup_trial]{setup_trial()}} and \code{\link[=run_trial]{run_trial()}}) will +be used instead of the posterior estimates.} \item{final_ests}{single logical. If \code{TRUE} (recommended) the final estimates calculated using outcome data from all patients randomised when trials are @@ -131,17 +131,21 @@ superiority. Will be \code{NA} in simulations not stopped for superiority. \item \code{selected_arm}: the final selected arm (as described above). Will correspond to the \code{superior_arm} in simulations stopped for superiority and be \code{NA} if no arm is selected. See \code{select_strategy} above. +\item \code{err}: the squared error of the estimate in the selected arm, +calculated as \verb{estimated effect - true effect} for the selected +arm. \item \verb{sq_err:} the squared error of the estimate in the selected arm, -calculated as \verb{(estimated effect - true effect)^2} for the selected -arms. -\item \code{sq_err_te}: the squared error of the treatment effect comparing -the selected arm to the comparator arm (as specified in \code{te_comp}). -Calculated as:\cr -\verb{((estimated effect in the selected arm - estimated effect in the comparator arm) -} -\verb{(true effect in the selected arm - true effect in the comparator arm))^2} \cr -Will be \code{NA} for simulations without a selected arm, with no +calculated as \code{err^2} for the selected arm, with \code{err} defined above. +\item \code{err_te}: the error of the treatment effect comparing the selected +arm to the comparator arm (as specified in \code{te_comp}). Calculated as:\cr +\verb{(estimated effect in the selected arm - estimated effect in the comparator arm) -} +\verb{(true effect in the selected arm - true effect in the comparator arm)} +\cr Will be \code{NA} for simulations without a selected arm, with no comparator specified (see \code{te_comp} above), and when the selected arm is the comparator arm. +\item \code{sq_err_te}: the squared error of the treatment effect comparing +the selected arm to the comparator arm (as specified in \code{te_comp}), +calculated as \code{err_te^2}, with \code{err_te} defined above. } } \description{ diff --git a/man/extract_results_batch.Rd b/man/extract_results_batch.Rd index e8e7401..1c1391c 100644 --- a/man/extract_results_batch.Rd +++ b/man/extract_results_batch.Rd @@ -72,9 +72,9 @@ available in the trial.} \item{te_comp}{character string, treatment-effect comparator. Can be either \code{NULL} (the default) in which case the \strong{first} \code{control} arm is used for trial designs with a common control arm, or a string naming a single trial -\code{arm}. Will be used when calculating \code{sq_err_te} (the squared error of the -treatment effect comparing the selected arm to the comparator arm, as -described below).} +\code{arm}. Will be used when calculating \code{err_te} and \code{sq_err_te} (the error +and the squared error of the treatment effect comparing the selected arm to +the comparator arm, as described below).} \item{which_ests}{single character string, a combination of the \code{raw_ests} and \code{final_ests} arguments from \code{\link[=extract_results]{extract_results()}}.} diff --git a/man/figures/README-plot-1.png b/man/figures/README-plot-1.png index 2f381af..bffe48a 100644 Binary files a/man/figures/README-plot-1.png and b/man/figures/README-plot-1.png differ diff --git a/man/figures/README-plot-2.png b/man/figures/README-plot-2.png index 4a73c75..515ba19 100644 Binary files a/man/figures/README-plot-2.png and b/man/figures/README-plot-2.png differ diff --git a/man/figures/README-unnamed-chunk-10-1.png b/man/figures/README-unnamed-chunk-10-1.png new file mode 100644 index 0000000..ef72fc8 Binary files /dev/null and b/man/figures/README-unnamed-chunk-10-1.png differ diff --git a/man/figures/README-unnamed-chunk-11-1.png b/man/figures/README-unnamed-chunk-11-1.png new file mode 100644 index 0000000..f08e460 Binary files /dev/null and b/man/figures/README-unnamed-chunk-11-1.png differ diff --git a/man/figures/README-unnamed-chunk-12-1.png b/man/figures/README-unnamed-chunk-12-1.png new file mode 100644 index 0000000..cf7beba Binary files /dev/null and b/man/figures/README-unnamed-chunk-12-1.png differ diff --git a/man/figures/README-unnamed-chunk-14-1.png b/man/figures/README-unnamed-chunk-14-1.png new file mode 100644 index 0000000..2cafd95 Binary files /dev/null and b/man/figures/README-unnamed-chunk-14-1.png differ diff --git a/man/figures/README-unnamed-chunk-15-1.png b/man/figures/README-unnamed-chunk-15-1.png new file mode 100644 index 0000000..8ad1454 Binary files /dev/null and b/man/figures/README-unnamed-chunk-15-1.png differ diff --git a/man/figures/README-unnamed-chunk-9-1.png b/man/figures/README-unnamed-chunk-9-1.png new file mode 100644 index 0000000..27f13eb Binary files /dev/null and b/man/figures/README-unnamed-chunk-9-1.png differ diff --git a/man/plot_convergence.Rd b/man/plot_convergence.Rd index 4158f8f..f695a17 100644 --- a/man/plot_convergence.Rd +++ b/man/plot_convergence.Rd @@ -34,9 +34,9 @@ Valid metrics include: \code{size_mean}, \code{size_sd}, \code{size_median}, \co \code{ratio_ys_p75}, \code{ratio_ys_p0}, \code{ratio_ys_p100}, \code{prob_conclusive}, \code{prob_superior}, \code{prob_equivalence}, \code{prob_futility}, \code{prob_max}, \verb{prob_select_*} (with \code{*} being either "\verb{arm_} for all \code{arm} names or -\code{none}), \code{rmse}, \code{rmse_te}, and \code{idp}. All may be specified as above, -case sensitive, but with either spaces or underlines. Defaults to -\code{"size mean"}.} +\code{none}), \code{rmse}, \code{rmse_te}, \code{mae}, \code{mae_te}, and \code{idp}. All may be +specified as above, case sensitive, but with either spaces or underlines. +Defaults to \code{"size mean"}.} \item{resolution}{single positive integer, the number of points calculated and plotted, defaults to \code{100} and must be \verb{>= 10}. Higher numbers lead to @@ -94,19 +94,19 @@ available in the trial.} \item{te_comp}{character string, treatment-effect comparator. Can be either \code{NULL} (the default) in which case the \strong{first} \code{control} arm is used for trial designs with a common control arm, or a string naming a single trial -\code{arm}. Will be used when calculating \code{sq_err_te} (the squared error of the -treatment effect comparing the selected arm to the comparator arm, as -described below).} +\code{arm}. Will be used when calculating \code{err_te} and \code{sq_err_te} (the error +and the squared error of the treatment effect comparing the selected arm to +the comparator arm, as described below).} \item{raw_ests}{single logical. If \code{FALSE} (default), the posterior estimates (\code{post_ests} or \code{post_ests_all}, see \code{\link[=setup_trial]{setup_trial()}} -and \code{\link[=run_trial]{run_trial()}}) will be used to calculate \code{sq_err} (the squared error of -the estimated compared to the specified effect in the selected arm) and -\code{sq_err_te} (the squared error of the treatment effect comparing the -selected arm to the comparator arm, as described for \code{te_comp} and below). -If \code{TRUE}, the raw estimates (\code{raw_ests} or \code{raw_ests_all}, see -\code{\link[=setup_trial]{setup_trial()}} and \code{\link[=run_trial]{run_trial()}}) will be used instead of the posterior -estimates.} +and \code{\link[=run_trial]{run_trial()}}) will be used to calculate \code{err} and \code{sq_err} (the error +and the squared error of the estimated compared to the specified effect in +the selected arm) and \code{err_te} and \code{sq_err_te} (the error and the squared +error of the treatment effect comparing the selected arm to the comparator +arm, as described for \code{te_comp} and below). If \code{TRUE}, the raw estimates +(\code{raw_ests} or \code{raw_ests_all}, see \code{\link[=setup_trial]{setup_trial()}} and \code{\link[=run_trial]{run_trial()}}) will +be used instead of the posterior estimates.} \item{final_ests}{single logical. If \code{TRUE} (recommended) the final estimates calculated using outcome data from all patients randomised when trials are diff --git a/man/plot_metrics_ecdf.Rd b/man/plot_metrics_ecdf.Rd index fea4e52..c960b91 100644 --- a/man/plot_metrics_ecdf.Rd +++ b/man/plot_metrics_ecdf.Rd @@ -7,6 +7,12 @@ plot_metrics_ecdf( object, metrics = c("size", "sum_ys", "ratio_ys"), + select_strategy = "control if available", + select_last_arm = FALSE, + select_preferences = NULL, + te_comp = NULL, + raw_ests = FALSE, + final_ests = NULL, restrict = NULL, nrow = NULL, ncol = NULL, @@ -19,9 +25,90 @@ function.} \item{metrics}{the performance metrics to plot, as described in \code{\link[=extract_results]{extract_results()}}. Multiple metrics may be plotted at the same time. -Valid metrics include: \code{size}, \code{sum_ys}, and \code{ratio_ys_mean}. All may be -specified using either spaces or underlines (case sensitive). Defaults to -plotting all three.} +Valid metrics include: \code{size}, \code{sum_ys}, \code{ratio_ys_mean}, \code{sq_err}, +\code{sq_err_te}, \code{err}, \code{err_te}, \code{abs_err}, \code{abs_err_te}, (as described in +\code{\link[=extract_results]{extract_results()}}, with the addition of \code{abs_err} and \code{abs_err_te}, which +are the absolute errors, i.e., \code{abs(err)} and \code{abs(err_te)}). All +may be specified using either spaces or underlines (case sensitive). +Defaults to plotting \code{size}, \code{sum_ys}, and \code{ratio_ys_mean}.} + +\item{select_strategy}{single character string. If a trial was not stopped +due to superiority (or had only 1 arm remaining, if \code{select_last_arm} is +set to \code{TRUE} in trial designs with a common \code{control} arm; see below), +this parameter specifies which arm will be considered selected when +calculating trial design performance metrics, as described below; +this corresponds to the consequence of an inconclusive trial, i.e., which +arm would then be used in practice.\cr +The following options are available and must be written exactly as below +(case sensitive, cannot be abbreviated): +\itemize{ +\item \code{"control if available"} (default): selects the \strong{first} +\code{control} arm for trials with a common \code{control} arm \emph{\strong{if}} this +arm is active at end-of-trial, otherwise no arm will be selected. For +trial designs without a common \code{control}, no arm will be selected. +\item \code{"none"}: selects no arm in trials not ending with superiority. +\item \code{"control"}: similar to \code{"control if available"}, but will throw +an error if used for trial designs without a common \code{control} arm. +\item \code{"final control"}: selects the \strong{final} \code{control} arm regardless +of whether the trial was stopped for practical equivalence, futility, +or at the maximum sample size; this strategy can only be specified +for trial designs with a common \code{control} arm. +\item \code{"control or best"}: selects the \strong{first} \code{control} arm if still +active at end-of-trial, otherwise selects the best remaining arm +(defined as the remaining arm with the highest probability of being +the best in the last adaptive analysis conducted). Only works for +trial designs with a common \code{control} arm. +\item \code{"best"}: selects the best remaining arm (as described under +\code{"control or best"}). +\item \code{"list or best"}: selects the first remaining arm from a specified +list (specified using \code{select_preferences}, technically a character +vector). If none of these arms are are active at end-of-trial, the best +remaining arm will be selected (as described above). +\item \code{"list"}: as specified above, but if no arms on the provided list +remain active at end-of-trial, no arm is selected. +}} + +\item{select_last_arm}{single logical, defaults to \code{FALSE}. If \code{TRUE}, the +only remaining active arm (the last \code{control}) will be selected in trials +with a common \code{control} arm ending with \code{equivalence} or \code{futility}, before +considering the options specified in \code{select_strategy}. Must be \code{FALSE} for +trial designs without a common \code{control} arm.} + +\item{select_preferences}{character vector specifying a number of arms used +for selection if one of the \code{"list or best"} or \code{"list"} options are +specified for \code{select_strategy}. Can only contain valid \code{arms} +available in the trial.} + +\item{te_comp}{character string, treatment-effect comparator. Can be either +\code{NULL} (the default) in which case the \strong{first} \code{control} arm is used for +trial designs with a common control arm, or a string naming a single trial +\code{arm}. Will be used when calculating \code{err_te} and \code{sq_err_te} (the error +and the squared error of the treatment effect comparing the selected arm to +the comparator arm, as described below).} + +\item{raw_ests}{single logical. If \code{FALSE} (default), the +posterior estimates (\code{post_ests} or \code{post_ests_all}, see \code{\link[=setup_trial]{setup_trial()}} +and \code{\link[=run_trial]{run_trial()}}) will be used to calculate \code{err} and \code{sq_err} (the error +and the squared error of the estimated compared to the specified effect in +the selected arm) and \code{err_te} and \code{sq_err_te} (the error and the squared +error of the treatment effect comparing the selected arm to the comparator +arm, as described for \code{te_comp} and below). If \code{TRUE}, the raw estimates +(\code{raw_ests} or \code{raw_ests_all}, see \code{\link[=setup_trial]{setup_trial()}} and \code{\link[=run_trial]{run_trial()}}) will +be used instead of the posterior estimates.} + +\item{final_ests}{single logical. If \code{TRUE} (recommended) the final estimates +calculated using outcome data from all patients randomised when trials are +stopped are used (\code{post_ests_all} or \code{raw_ests_all}, see \code{\link[=setup_trial]{setup_trial()}} +and \code{\link[=run_trial]{run_trial()}}); if \code{FALSE}, the estimates calculated for each arm when +an arm is stopped (or at the last adaptive analysis if not before) using +data from patients having reach followed up at this time point and not all +patients randomised are used (\code{post_ests} or \code{raw_ests}, see +\code{\link[=setup_trial]{setup_trial()}} and \code{\link[=run_trial]{run_trial()}}). If \code{NULL} (the default), this argument +will be set to \code{FALSE} if outcome data are available immediate after +randomisation for all patients (for backwards compatibility, as final +posterior estimates may vary slightly in this situation, even if using the +same data); otherwise it will be said to \code{TRUE}. See \code{\link[=setup_trial]{setup_trial()}} for +more details on how these estimates are calculated.} \item{restrict}{single character string or \code{NULL}. If \code{NULL} (default), results are summarised for all simulations; if \code{"superior"}, results are @@ -56,6 +143,10 @@ Plots empirical cumulative distribution functions (ECDFs) of numerical performance metrics across multiple simulations from a \code{"trial_results"} object returned by \code{\link[=run_trials]{run_trials()}}. Requires the \code{ggplot2} package installed. } +\details{ +Note that the arguments related to arm selection and error calculation are +only relevant if errors are visualised. +} \examples{ #### Only run examples if ggplot2 is installed #### if (requireNamespace("ggplot2", quietly = TRUE)){ diff --git a/man/print.Rd b/man/print.Rd index e93a5f8..6e1b29f 100644 --- a/man/print.Rd +++ b/man/print.Rd @@ -100,19 +100,19 @@ available in the trial.} \item{te_comp}{character string, treatment-effect comparator. Can be either \code{NULL} (the default) in which case the \strong{first} \code{control} arm is used for trial designs with a common control arm, or a string naming a single trial -\code{arm}. Will be used when calculating \code{sq_err_te} (the squared error of the -treatment effect comparing the selected arm to the comparator arm, as -described below).} +\code{arm}. Will be used when calculating \code{err_te} and \code{sq_err_te} (the error +and the squared error of the treatment effect comparing the selected arm to +the comparator arm, as described below).} \item{raw_ests}{single logical. If \code{FALSE} (default), the posterior estimates (\code{post_ests} or \code{post_ests_all}, see \code{\link[=setup_trial]{setup_trial()}} -and \code{\link[=run_trial]{run_trial()}}) will be used to calculate \code{sq_err} (the squared error of -the estimated compared to the specified effect in the selected arm) and -\code{sq_err_te} (the squared error of the treatment effect comparing the -selected arm to the comparator arm, as described for \code{te_comp} and below). -If \code{TRUE}, the raw estimates (\code{raw_ests} or \code{raw_ests_all}, see -\code{\link[=setup_trial]{setup_trial()}} and \code{\link[=run_trial]{run_trial()}}) will be used instead of the posterior -estimates.} +and \code{\link[=run_trial]{run_trial()}}) will be used to calculate \code{err} and \code{sq_err} (the error +and the squared error of the estimated compared to the specified effect in +the selected arm) and \code{err_te} and \code{sq_err_te} (the error and the squared +error of the treatment effect comparing the selected arm to the comparator +arm, as described for \code{te_comp} and below). If \code{TRUE}, the raw estimates +(\code{raw_ests} or \code{raw_ests_all}, see \code{\link[=setup_trial]{setup_trial()}} and \code{\link[=run_trial]{run_trial()}}) will +be used instead of the posterior estimates.} \item{final_ests}{single logical. If \code{TRUE} (recommended) the final estimates calculated using outcome data from all patients randomised when trials are diff --git a/man/reallocate_probs.Rd b/man/reallocate_probs.Rd index 95942cd..61cdea8 100644 --- a/man/reallocate_probs.Rd +++ b/man/reallocate_probs.Rd @@ -10,7 +10,11 @@ reallocate_probs( min_probs, max_probs, soften_power = 1, - match_arm = NULL + match_arm = NULL, + rescale_fixed = FALSE, + rescale_limits = FALSE, + rescale_factor = 1, + rescale_ignore = NULL ) } \arguments{ @@ -46,6 +50,18 @@ then no \emph{softening} is applied.} \item{match_arm}{index of the \code{control} arm. If not \code{NULL} (default), the control arm allocation probability will be similar to that of the best non-control arm. Must be \code{NULL} in designs without a common control arm.} + +\item{rescale_fixed}{logical indicating whether \code{fixed_probs} should be +rescaled following arm dropping.} + +\item{rescale_limits}{logical indicating whether \code{min/max_probs} should be +rescaled following arm dropping.} + +\item{rescale_factor}{numerical, rescale factor defined as +\verb{initial number of arms/number of active arms}.} + +\item{rescale_ignore}{\code{NULL} or index of an arm that will be ignored by the +\code{rescale_fixed} and \code{rescale_limits} arguments.} } \value{ A named (according to the \code{arms}) numeric vector with updated diff --git a/man/setup_trial.Rd b/man/setup_trial.Rd index bafba01..39775b1 100644 --- a/man/setup_trial.Rd +++ b/man/setup_trial.Rd @@ -13,6 +13,7 @@ setup_trial( fixed_probs = NULL, min_probs = rep(NA, length(arms)), max_probs = rep(NA, length(arms)), + rescale_probs = NULL, data_looks = NULL, max_n = NULL, look_after_every = NULL, @@ -75,6 +76,30 @@ probabilities; higher probabilities will be rounded down to these values. Must be \code{NA} (default for all arms) if no threshold is wanted and for arms using fixed allocation probabilities.} +\item{rescale_probs}{\code{NULL} (default) or one of either \code{"fixed"}, \code{"limits"}, +or \code{"both"}. Rescales \code{fixed_probs} (if \code{"fixed"} or \code{"both"}) and +\code{min_probs/max_probs} (if \code{"limits"} or \code{"both"}) after arm dropping in +trial specifications with \verb{>2 arms} using a \code{rescale_factor} defined as +\verb{initial number of arms/number of active arms}. \verb{"fixed_probs} and +\code{min_probs} are rescaled as \verb{initial value * rescale factor}, except for +\code{fixed_probs} controlled by the \code{control_prob_fixed} argument, which are +never rescaled. \code{max_probs} are rescaled as +\verb{1 - ( (1 - initial value) * rescale_factor)}.\cr +Must be \code{NULL} if there are only \verb{2 arms} or if \code{control_prob_fixed} is +\code{"sqrt-based fixed"}. If not \code{NULL}, one or more valid non-\code{NA} values must +be specified for either \code{min_probs/max_probs} or \code{fixed_probs} (not +counting a fixed value for the original \code{control} if \code{control_prob_fixed} +is \code{"sqrt-based"/"sqrt-based start"/"sqrt-based fixed"}).\cr +\strong{Note:} using this argument and specific combinations of values in +the other arguments may lead to invalid combined (total) allocation +probabilities after arm dropping, in which case all probabilities will +ultimately be rescaled to sum to \code{1}. It is the responsibility of the user +to ensure that rescaling fixed allocation probabilities and minimum/maximum +allocation probability limits will not lead to invalid or unexpected +allocation probabilities after arm dropping.\cr +Finally, any initial values that are overwritten by the +\code{control_prob_fixed} argument after arm dropping will not be rescaled.} + \item{data_looks}{vector of increasing integers, specifies when to conduct adaptive analyses (= the total number of patients with available outcome data at each adaptive analysis). The last number in the vector represents @@ -120,10 +145,11 @@ behaviour with respect to these comparisons.} be set \code{NULL} (the default), in which case the control arm allocation probability will not be fixed if control arms change (the allocation probability for the first control arm may still be fixed using -\code{fixed_probs}). If not \code{NULL}, a vector of probabilities of either length -\code{1} or \verb{number of arms - 1} can be provided, or one of the special -arguments \code{"sqrt-based"}, \code{"sqrt-based start"}, \code{"sqrt-based fixed"} or -\code{"match"}. See \code{\link[=setup_trial]{setup_trial()}} \strong{Details} for details on how this affects +\code{fixed_probs}, but will not be 'reused' for the new control arm).\cr +If not \code{NULL}, a vector of probabilities of either length \code{1} or +\verb{number of arms - 1} can be provided, or one of the special arguments +\code{"sqrt-based"}, \code{"sqrt-based start"}, \code{"sqrt-based fixed"} or \code{"match"}.\cr +See \code{\link[=setup_trial]{setup_trial()}} \strong{Details} for details on how this affects trial behaviour.} \item{inferiority}{single numeric value or vector of numeric values of the @@ -349,8 +375,8 @@ If the \code{fun_y_gen}, \code{fun_draws}, or \code{fun_raw_est} functions calls user-specified functions (or uses objects defined by the user outside these functions or the \code{\link[=setup_trial]{setup_trial()}}-call) or functions from external packages and simulations are conducted on multiple cores, these objects or functions -must be exported or prefixed with their namespaces, respectively, as -described in \code{\link[=setup_cluster]{setup_cluster()}} and \code{\link[=run_trials]{run_trials()}}. +must be prefixed with their namespaces (i.e., \verb{package::function()}) or +exported, as described in \code{\link[=setup_cluster]{setup_cluster()}} and \code{\link[=run_trials]{run_trials()}}. \strong{More information on arguments} \itemize{ @@ -381,14 +407,17 @@ allocation probabilities. These are defined as:\cr scaled to sum to 1, which will generally increase power for comparisons against the common \code{control}, as discussed in, e.g., \emph{Park et al, 2020} \doi{10.1016/j.jclinepi.2020.04.025}.\cr -If \code{"sqrt-based"}, square-root-transformation-based allocation probabilities -will also be used for new controls when arms are dropped. If -\code{"sqrt-based start"}, the control arm will be fixed to this allocation -probability at all times (also after arm dropping, with rescaling as -necessary, as specified above). If \code{"sqrt-based fixed"} is chosen, -square-root-transformation-based allocation probabilities will be used and -all allocation probabilities will be fixed throughout the trial (with -rescaling when arms are dropped).\cr +If \code{"sqrt-based"} or \code{"sqrt-based fixed"}, square-root-transformation-based +allocation probabilities will be used initially and also for new controls +when arms are dropped (with probabilities always calculated based on the +number of active non-control arms). If \code{"sqrt-based"}, response-adaptive +randomisation will be used for non-control arms, while the non-control arms +will use fixed, square-root based allocation probabilities at all times (with +probabilities always calculated based on the number of active non-control +arms). If \code{"sqrt-based start"}, the control arm allocation probability will +be fixed to a square-root based probability at all times calculated according +to the initial number of arms (with this probability also being used for new +control(s) when the original control is dropped).\cr If \code{"match"} is specified, the control group allocation probability will always be \emph{matched} to be similar to the highest non-control arm allocation probability. diff --git a/man/setup_trial_binom.Rd b/man/setup_trial_binom.Rd index 542a485..f21848c 100644 --- a/man/setup_trial_binom.Rd +++ b/man/setup_trial_binom.Rd @@ -11,6 +11,7 @@ setup_trial_binom( fixed_probs = NULL, min_probs = rep(NA, length(arms)), max_probs = rep(NA, length(arms)), + rescale_probs = NULL, data_looks = NULL, max_n = NULL, look_after_every = NULL, @@ -61,6 +62,30 @@ probabilities; higher probabilities will be rounded down to these values. Must be \code{NA} (default for all arms) if no threshold is wanted and for arms using fixed allocation probabilities.} +\item{rescale_probs}{\code{NULL} (default) or one of either \code{"fixed"}, \code{"limits"}, +or \code{"both"}. Rescales \code{fixed_probs} (if \code{"fixed"} or \code{"both"}) and +\code{min_probs/max_probs} (if \code{"limits"} or \code{"both"}) after arm dropping in +trial specifications with \verb{>2 arms} using a \code{rescale_factor} defined as +\verb{initial number of arms/number of active arms}. \verb{"fixed_probs} and +\code{min_probs} are rescaled as \verb{initial value * rescale factor}, except for +\code{fixed_probs} controlled by the \code{control_prob_fixed} argument, which are +never rescaled. \code{max_probs} are rescaled as +\verb{1 - ( (1 - initial value) * rescale_factor)}.\cr +Must be \code{NULL} if there are only \verb{2 arms} or if \code{control_prob_fixed} is +\code{"sqrt-based fixed"}. If not \code{NULL}, one or more valid non-\code{NA} values must +be specified for either \code{min_probs/max_probs} or \code{fixed_probs} (not +counting a fixed value for the original \code{control} if \code{control_prob_fixed} +is \code{"sqrt-based"/"sqrt-based start"/"sqrt-based fixed"}).\cr +\strong{Note:} using this argument and specific combinations of values in +the other arguments may lead to invalid combined (total) allocation +probabilities after arm dropping, in which case all probabilities will +ultimately be rescaled to sum to \code{1}. It is the responsibility of the user +to ensure that rescaling fixed allocation probabilities and minimum/maximum +allocation probability limits will not lead to invalid or unexpected +allocation probabilities after arm dropping.\cr +Finally, any initial values that are overwritten by the +\code{control_prob_fixed} argument after arm dropping will not be rescaled.} + \item{data_looks}{vector of increasing integers, specifies when to conduct adaptive analyses (= the total number of patients with available outcome data at each adaptive analysis). The last number in the vector represents @@ -106,10 +131,11 @@ behaviour with respect to these comparisons.} be set \code{NULL} (the default), in which case the control arm allocation probability will not be fixed if control arms change (the allocation probability for the first control arm may still be fixed using -\code{fixed_probs}). If not \code{NULL}, a vector of probabilities of either length -\code{1} or \verb{number of arms - 1} can be provided, or one of the special -arguments \code{"sqrt-based"}, \code{"sqrt-based start"}, \code{"sqrt-based fixed"} or -\code{"match"}. See \code{\link[=setup_trial]{setup_trial()}} \strong{Details} for details on how this affects +\code{fixed_probs}, but will not be 'reused' for the new control arm).\cr +If not \code{NULL}, a vector of probabilities of either length \code{1} or +\verb{number of arms - 1} can be provided, or one of the special arguments +\code{"sqrt-based"}, \code{"sqrt-based start"}, \code{"sqrt-based fixed"} or \code{"match"}.\cr +See \code{\link[=setup_trial]{setup_trial()}} \strong{Details} for details on how this affects trial behaviour.} \item{inferiority}{single numeric value or vector of numeric values of the diff --git a/man/setup_trial_norm.Rd b/man/setup_trial_norm.Rd index cc276bf..1566769 100644 --- a/man/setup_trial_norm.Rd +++ b/man/setup_trial_norm.Rd @@ -12,6 +12,7 @@ setup_trial_norm( fixed_probs = NULL, min_probs = rep(NA, length(arms)), max_probs = rep(NA, length(arms)), + rescale_probs = NULL, data_looks = NULL, max_n = NULL, look_after_every = NULL, @@ -65,6 +66,30 @@ probabilities; higher probabilities will be rounded down to these values. Must be \code{NA} (default for all arms) if no threshold is wanted and for arms using fixed allocation probabilities.} +\item{rescale_probs}{\code{NULL} (default) or one of either \code{"fixed"}, \code{"limits"}, +or \code{"both"}. Rescales \code{fixed_probs} (if \code{"fixed"} or \code{"both"}) and +\code{min_probs/max_probs} (if \code{"limits"} or \code{"both"}) after arm dropping in +trial specifications with \verb{>2 arms} using a \code{rescale_factor} defined as +\verb{initial number of arms/number of active arms}. \verb{"fixed_probs} and +\code{min_probs} are rescaled as \verb{initial value * rescale factor}, except for +\code{fixed_probs} controlled by the \code{control_prob_fixed} argument, which are +never rescaled. \code{max_probs} are rescaled as +\verb{1 - ( (1 - initial value) * rescale_factor)}.\cr +Must be \code{NULL} if there are only \verb{2 arms} or if \code{control_prob_fixed} is +\code{"sqrt-based fixed"}. If not \code{NULL}, one or more valid non-\code{NA} values must +be specified for either \code{min_probs/max_probs} or \code{fixed_probs} (not +counting a fixed value for the original \code{control} if \code{control_prob_fixed} +is \code{"sqrt-based"/"sqrt-based start"/"sqrt-based fixed"}).\cr +\strong{Note:} using this argument and specific combinations of values in +the other arguments may lead to invalid combined (total) allocation +probabilities after arm dropping, in which case all probabilities will +ultimately be rescaled to sum to \code{1}. It is the responsibility of the user +to ensure that rescaling fixed allocation probabilities and minimum/maximum +allocation probability limits will not lead to invalid or unexpected +allocation probabilities after arm dropping.\cr +Finally, any initial values that are overwritten by the +\code{control_prob_fixed} argument after arm dropping will not be rescaled.} + \item{data_looks}{vector of increasing integers, specifies when to conduct adaptive analyses (= the total number of patients with available outcome data at each adaptive analysis). The last number in the vector represents @@ -110,10 +135,11 @@ behaviour with respect to these comparisons.} be set \code{NULL} (the default), in which case the control arm allocation probability will not be fixed if control arms change (the allocation probability for the first control arm may still be fixed using -\code{fixed_probs}). If not \code{NULL}, a vector of probabilities of either length -\code{1} or \verb{number of arms - 1} can be provided, or one of the special -arguments \code{"sqrt-based"}, \code{"sqrt-based start"}, \code{"sqrt-based fixed"} or -\code{"match"}. See \code{\link[=setup_trial]{setup_trial()}} \strong{Details} for details on how this affects +\code{fixed_probs}, but will not be 'reused' for the new control arm).\cr +If not \code{NULL}, a vector of probabilities of either length \code{1} or +\verb{number of arms - 1} can be provided, or one of the special arguments +\code{"sqrt-based"}, \code{"sqrt-based start"}, \code{"sqrt-based fixed"} or \code{"match"}.\cr +See \code{\link[=setup_trial]{setup_trial()}} \strong{Details} for details on how this affects trial behaviour.} \item{inferiority}{single numeric value or vector of numeric values of the diff --git a/man/summary.Rd b/man/summary.Rd index 337e0e6..cc2de15 100644 --- a/man/summary.Rd +++ b/man/summary.Rd @@ -72,19 +72,19 @@ available in the trial.} \item{te_comp}{character string, treatment-effect comparator. Can be either \code{NULL} (the default) in which case the \strong{first} \code{control} arm is used for trial designs with a common control arm, or a string naming a single trial -\code{arm}. Will be used when calculating \code{sq_err_te} (the squared error of the -treatment effect comparing the selected arm to the comparator arm, as -described below).} +\code{arm}. Will be used when calculating \code{err_te} and \code{sq_err_te} (the error +and the squared error of the treatment effect comparing the selected arm to +the comparator arm, as described below).} \item{raw_ests}{single logical. If \code{FALSE} (default), the posterior estimates (\code{post_ests} or \code{post_ests_all}, see \code{\link[=setup_trial]{setup_trial()}} -and \code{\link[=run_trial]{run_trial()}}) will be used to calculate \code{sq_err} (the squared error of -the estimated compared to the specified effect in the selected arm) and -\code{sq_err_te} (the squared error of the treatment effect comparing the -selected arm to the comparator arm, as described for \code{te_comp} and below). -If \code{TRUE}, the raw estimates (\code{raw_ests} or \code{raw_ests_all}, see -\code{\link[=setup_trial]{setup_trial()}} and \code{\link[=run_trial]{run_trial()}}) will be used instead of the posterior -estimates.} +and \code{\link[=run_trial]{run_trial()}}) will be used to calculate \code{err} and \code{sq_err} (the error +and the squared error of the estimated compared to the specified effect in +the selected arm) and \code{err_te} and \code{sq_err_te} (the error and the squared +error of the treatment effect comparing the selected arm to the comparator +arm, as described for \code{te_comp} and below). If \code{TRUE}, the raw estimates +(\code{raw_ests} or \code{raw_ests_all}, see \code{\link[=setup_trial]{setup_trial()}} and \code{\link[=run_trial]{run_trial()}}) will +be used instead of the posterior estimates.} \item{final_ests}{single logical. If \code{TRUE} (recommended) the final estimates calculated using outcome data from all patients randomised when trials are @@ -137,11 +137,11 @@ A \code{"trial_results_summary"} object containing the following values: \code{ratio_ys_p0}, \code{ratio_ys_p100}, \code{prob_conclusive}, \code{prob_superior}, \code{prob_equivalence}, \code{prob_futility}, \code{prob_max}, \verb{prob_select_*} (with \code{*} being either "\verb{arm_} for all \code{arm} names or \code{none}), \code{rmse}, -\code{rmse_te}, and \code{idp}: performance metrics as described in -\code{\link[=check_performance]{check_performance()}}. Note that all \code{sum_ys_} and \code{ratio_ys_} measures -use outcome data from all randomised patients, regardless of whether they -had outcome data available at the last analysis or not, as described in -\code{\link[=extract_results]{extract_results()}}. +\code{rmse_te}, \code{mae}, \code{mae_te}, and \code{idp}: performance metrics as described +in \code{\link[=check_performance]{check_performance()}}. Note that all \code{sum_ys_} and \code{ratio_ys_} +measures use outcome data from all randomised patients, regardless of +whether they had outcome data available at the last analysis or not, as +described in \code{\link[=extract_results]{extract_results()}}. \item \code{select_strategy}, \code{select_last_arm}, \code{select_preferences}, \code{te_comp}, \code{raw_ests}, \code{final_ests}, \code{restrict}: as specified above. \item \code{control}: the control arm specified by \code{\link[=setup_trial]{setup_trial()}}, diff --git a/man/update_saved_calibration.Rd b/man/update_saved_calibration.Rd new file mode 100644 index 0000000..4cbe9e1 --- /dev/null +++ b/man/update_saved_calibration.Rd @@ -0,0 +1,49 @@ +% Generated by roxygen2: do not edit by hand +% Please edit documentation in R/update_saved.R +\name{update_saved_calibration} +\alias{update_saved_calibration} +\title{Update previously saved calibration result} +\usage{ +update_saved_calibration(path, version = NULL, compress = TRUE) +} +\arguments{ +\item{path}{single character; the path to the saved +\code{"trial_calibration"}-object containing the calibration result saved by +\code{\link[=calibrate_trial]{calibrate_trial()}}.} + +\item{version}{passed to \code{\link[=saveRDS]{saveRDS()}} when saving the updated object, defaults +to \code{NULL} (as in \code{\link[=saveRDS]{saveRDS()}}), which means that the current default version +is used.} + +\item{compress}{passed to \code{\link[=saveRDS]{saveRDS()}} when saving the updated object, +defaults to \code{TRUE} (as in \code{\link[=saveRDS]{saveRDS()}}), see \code{\link[=saveRDS]{saveRDS()}} for other options.} +} +\value{ +Invisibly returns the updated \code{"trial_calibration"}-object. +} +\description{ +This function updates a previously saved \code{"trial_calibration"}-object created +and saved by \code{\link[=calibrate_trial]{calibrate_trial()}} using a previous version of \code{adaptr}, +including the embedded trial specification and trial results objects +(internally using the \code{\link[=update_saved_trials]{update_saved_trials()}} function). This allows the +use of calibration results, including the calibrated trial specification and +the best simulations results from the calibration process, to be used without +errors by this version of the package. The function should be run only once +per saved simulation object and will issue a warning if the object is already +up to date. And overview of the changes made according to the \code{adaptr} package +version used to generate the original object is provided in \strong{Details}.\cr +} +\details{ +The following changes are made according to the version of \code{adaptr} used to +generate the original \code{"trial_calibration"} object: +\itemize{ +\item \verb{v1.3.0+}: updates version number of the +\code{"trial_calibration"}-object and updates the embedded +\code{"trial_results"}-object (saved in \verb{$best_sims}, if any) and +\code{"trial_spec"}-objects (saved in \verb{$input_trial_spec} and +\verb{$best_trial_spec}) as described in \code{\link[=update_saved_trials]{update_saved_trials()}}. +} +} +\seealso{ +\code{\link[=run_trials]{run_trials()}}. +} diff --git a/man/update_saved_trials.Rd b/man/update_saved_trials.Rd index 88f8651..10a9555 100644 --- a/man/update_saved_trials.Rd +++ b/man/update_saved_trials.Rd @@ -1,5 +1,5 @@ % Generated by roxygen2: do not edit by hand -% Please edit documentation in R/update_saved_trials.R +% Please edit documentation in R/update_saved.R \name{update_saved_trials} \alias{update_saved_trials} \title{Update previously saved simulation results} @@ -48,12 +48,14 @@ version of the package. The following changes are made according to the version of \code{adaptr} used to generate the original \code{"trial_results"} object: \itemize{ -\item \verb{v1.2.0+}: only updates the version number. +\item \verb{v1.2.0+}: updates version number and the \code{reallocate_probs} +argument in the embedded trial specification. \item \verb{v1.1.1 or earlier}: updates version number and everything related to follow-up and data collection lag (in these versions, the \code{randomised_at_looks} argument in the \code{\link[=setup_trial]{setup_trial()}} functions did not exist, but for practical purposes was identical to the number of -patients with available data at each look). +patients with available data at each look) and the \code{reallocate_probs} +argument in the embedded trial specification. } } \seealso{ diff --git a/man/validate_trial.Rd b/man/validate_trial.Rd index 4ac44d7..fa9c25e 100644 --- a/man/validate_trial.Rd +++ b/man/validate_trial.Rd @@ -11,6 +11,7 @@ validate_trial( fixed_probs = NULL, min_probs = rep(NA, length(arms)), max_probs = rep(NA, length(arms)), + rescale_probs = NULL, data_looks = NULL, max_n = NULL, look_after_every = NULL, @@ -65,6 +66,30 @@ probabilities; higher probabilities will be rounded down to these values. Must be \code{NA} (default for all arms) if no threshold is wanted and for arms using fixed allocation probabilities.} +\item{rescale_probs}{\code{NULL} (default) or one of either \code{"fixed"}, \code{"limits"}, +or \code{"both"}. Rescales \code{fixed_probs} (if \code{"fixed"} or \code{"both"}) and +\code{min_probs/max_probs} (if \code{"limits"} or \code{"both"}) after arm dropping in +trial specifications with \verb{>2 arms} using a \code{rescale_factor} defined as +\verb{initial number of arms/number of active arms}. \verb{"fixed_probs} and +\code{min_probs} are rescaled as \verb{initial value * rescale factor}, except for +\code{fixed_probs} controlled by the \code{control_prob_fixed} argument, which are +never rescaled. \code{max_probs} are rescaled as +\verb{1 - ( (1 - initial value) * rescale_factor)}.\cr +Must be \code{NULL} if there are only \verb{2 arms} or if \code{control_prob_fixed} is +\code{"sqrt-based fixed"}. If not \code{NULL}, one or more valid non-\code{NA} values must +be specified for either \code{min_probs/max_probs} or \code{fixed_probs} (not +counting a fixed value for the original \code{control} if \code{control_prob_fixed} +is \code{"sqrt-based"/"sqrt-based start"/"sqrt-based fixed"}).\cr +\strong{Note:} using this argument and specific combinations of values in +the other arguments may lead to invalid combined (total) allocation +probabilities after arm dropping, in which case all probabilities will +ultimately be rescaled to sum to \code{1}. It is the responsibility of the user +to ensure that rescaling fixed allocation probabilities and minimum/maximum +allocation probability limits will not lead to invalid or unexpected +allocation probabilities after arm dropping.\cr +Finally, any initial values that are overwritten by the +\code{control_prob_fixed} argument after arm dropping will not be rescaled.} + \item{data_looks}{vector of increasing integers, specifies when to conduct adaptive analyses (= the total number of patients with available outcome data at each adaptive analysis). The last number in the vector represents @@ -110,10 +135,11 @@ behaviour with respect to these comparisons.} be set \code{NULL} (the default), in which case the control arm allocation probability will not be fixed if control arms change (the allocation probability for the first control arm may still be fixed using -\code{fixed_probs}). If not \code{NULL}, a vector of probabilities of either length -\code{1} or \verb{number of arms - 1} can be provided, or one of the special -arguments \code{"sqrt-based"}, \code{"sqrt-based start"}, \code{"sqrt-based fixed"} or -\code{"match"}. See \code{\link[=setup_trial]{setup_trial()}} \strong{Details} for details on how this affects +\code{fixed_probs}, but will not be 'reused' for the new control arm).\cr +If not \code{NULL}, a vector of probabilities of either length \code{1} or +\verb{number of arms - 1} can be provided, or one of the special arguments +\code{"sqrt-based"}, \code{"sqrt-based start"}, \code{"sqrt-based fixed"} or \code{"match"}.\cr +See \code{\link[=setup_trial]{setup_trial()}} \strong{Details} for details on how this affects trial behaviour.} \item{inferiority}{single numeric value or vector of numeric values of the diff --git a/tests/testthat/_snaps/check_performance.md b/tests/testthat/_snaps/check_performance.md index 67b1997..8ad841a 100644 --- a/tests/testthat/_snaps/check_performance.md +++ b/tests/testthat/_snaps/check_performance.md @@ -37,7 +37,9 @@ 31 prob_select_none 0.350 32 rmse 0.020 33 rmse_te NA - 34 idp 100.000 + 34 mae 0.009 + 35 mae_te NA + 36 idp 100.000 --- @@ -78,13 +80,16 @@ 31 prob_select_none 0.350 32 rmse 0.020 33 rmse_te NA - 34 idp 100.000 + 34 mae 0.009 + 35 mae_te NA + 36 idp 100.000 --- Code check_performance(res, uncertainty = TRUE, boot_seed = 4131, n_boot = 100) - Warning + Condition + Warning: values for n_boot < 1000 are not recommended, as they may cause instable results. Output metric est err_sd err_mad lo_ci hi_ci @@ -121,14 +126,17 @@ 31 prob_select_none 0.350 0.103 0.111 0.200 0.550 32 rmse 0.020 0.004 0.003 0.011 0.026 33 rmse_te NA NA NA NA NA - 34 idp 100.000 0.000 0.000 100.000 100.000 + 34 mae 0.009 0.005 0.003 0.005 0.025 + 35 mae_te NA NA NA NA NA + 36 idp 100.000 0.000 0.000 100.000 100.000 --- Code check_performance(res, uncertainty = TRUE, ci_width = 0.75, boot_seed = 4131, n_boot = 100) - Warning + Condition + Warning: values for n_boot < 1000 are not recommended, as they may cause instable results. Output metric est err_sd err_mad lo_ci hi_ci @@ -165,7 +173,9 @@ 31 prob_select_none 0.350 0.103 0.111 0.250 0.500 32 rmse 0.020 0.004 0.003 0.014 0.023 33 rmse_te NA NA NA NA NA - 34 idp 100.000 0.000 0.000 100.000 100.000 + 34 mae 0.009 0.005 0.003 0.007 0.018 + 35 mae_te NA NA NA NA NA + 36 idp 100.000 0.000 0.000 100.000 100.000 --- @@ -206,7 +216,9 @@ 31 prob_select_none 0.000 32 rmse 0.020 33 rmse_te NA - 34 idp 100.000 + 34 mae 0.009 + 35 mae_te NA + 36 idp 100.000 --- @@ -247,14 +259,17 @@ 31 prob_select_none 0.000 32 rmse 0.020 33 rmse_te NA - 34 idp 100.000 + 34 mae 0.009 + 35 mae_te NA + 36 idp 100.000 --- Code check_performance(res, restrict = "superior", uncertainty = TRUE, boot_seed = "base", n_boot = 100) - Warning + Condition + Warning: values for n_boot < 1000 are not recommended, as they may cause instable results. Output metric est err_sd err_mad lo_ci hi_ci @@ -291,14 +306,17 @@ 31 prob_select_none 0.000 0.000 0.000 0.000 0.000 32 rmse 0.020 0.003 0.004 0.013 0.025 33 rmse_te NA NA NA NA NA - 34 idp 100.000 0.000 0.000 100.000 100.000 + 34 mae 0.009 0.005 0.003 0.006 0.025 + 35 mae_te NA NA NA NA NA + 36 idp 100.000 0.000 0.000 100.000 100.000 --- Code check_performance(res, restrict = "selected", uncertainty = TRUE, boot_seed = "base", n_boot = 100) - Warning + Condition + Warning: values for n_boot < 1000 are not recommended, as they may cause instable results. Output metric est err_sd err_mad lo_ci hi_ci @@ -335,5 +353,7 @@ 31 prob_select_none 0.000 0.000 0.000 0.000 0.000 32 rmse 0.020 0.003 0.004 0.013 0.025 33 rmse_te NA NA NA NA NA - 34 idp 100.000 0.000 0.000 100.000 100.000 + 34 mae 0.009 0.005 0.003 0.006 0.025 + 35 mae_te NA NA NA NA NA + 36 idp 100.000 0.000 0.000 100.000 100.000 diff --git a/tests/testthat/_snaps/extract.md b/tests/testthat/_snaps/extract.md index 3138031..a2040c1 100644 --- a/tests/testthat/_snaps/extract.md +++ b/tests/testthat/_snaps/extract.md @@ -24,27 +24,27 @@ 18 18 700 199 0.2842857 futility 19 19 1300 345 0.2653846 futility 20 20 300 86 0.2866667 futility - sq_err sq_err_te - 1 NA NA - 2 1.624546e-03 3.500275e-04 - 3 NA NA - 4 NA NA - 5 1.273993e-04 8.827508e-04 - 6 NA NA - 7 NA NA - 8 NA NA - 9 NA NA - 10 3.479209e-04 4.446702e-05 - 11 NA NA - 12 1.399986e-07 5.469797e-04 - 13 NA NA - 14 NA NA - 15 1.334537e-03 8.410892e-04 - 16 NA NA - 17 NA NA - 18 NA NA - 19 NA NA - 20 NA NA + err sq_err err_te sq_err_te + 1 NA NA NA NA + 2 0.0403056552 1.624546e-03 0.01870902 3.500275e-04 + 3 NA NA NA NA + 4 NA NA NA NA + 5 0.0112871320 1.273993e-04 -0.02971112 8.827508e-04 + 6 NA NA NA NA + 7 NA NA NA NA + 8 NA NA NA NA + 9 NA NA NA NA + 10 -0.0186526388 3.479209e-04 -0.00666836 4.446702e-05 + 11 NA NA NA NA + 12 0.0003741639 1.399986e-07 0.02338760 5.469797e-04 + 13 NA NA NA NA + 14 NA NA NA NA + 15 -0.0365313171 1.334537e-03 -0.02900154 8.410892e-04 + 16 NA NA NA NA + 17 NA NA NA NA + 18 NA NA NA NA + 19 NA NA NA NA + 20 NA NA NA NA --- @@ -72,27 +72,27 @@ 18 18 700 199 0.2842857 futility 19 19 1300 345 0.2653846 futility 20 20 300 86 0.2866667 futility - sq_err sq_err_te - 1 NA NA - 2 1.624546e-03 3.500275e-04 - 3 NA NA - 4 NA NA - 5 1.273993e-04 8.827508e-04 - 6 NA NA - 7 NA NA - 8 NA NA - 9 NA NA - 10 3.479209e-04 4.446702e-05 - 11 NA NA - 12 1.399986e-07 5.469797e-04 - 13 NA NA - 14 NA NA - 15 1.334537e-03 8.410892e-04 - 16 NA NA - 17 NA NA - 18 NA NA - 19 NA NA - 20 NA NA + err sq_err err_te sq_err_te + 1 NA NA NA NA + 2 0.0403056552 1.624546e-03 0.01870902 3.500275e-04 + 3 NA NA NA NA + 4 NA NA NA NA + 5 0.0112871320 1.273993e-04 -0.02971112 8.827508e-04 + 6 NA NA NA NA + 7 NA NA NA NA + 8 NA NA NA NA + 9 NA NA NA NA + 10 -0.0186526388 3.479209e-04 -0.00666836 4.446702e-05 + 11 NA NA NA NA + 12 0.0003741639 1.399986e-07 0.02338760 5.469797e-04 + 13 NA NA NA NA + 14 NA NA NA NA + 15 -0.0365313171 1.334537e-03 -0.02900154 8.410892e-04 + 16 NA NA NA NA + 17 NA NA NA NA + 18 NA NA NA NA + 19 NA NA NA NA + 20 NA NA NA NA --- @@ -120,27 +120,27 @@ 18 18 700 199 0.2842857 futility 19 19 1300 345 0.2653846 futility 20 20 300 86 0.2866667 futility - sq_err sq_err_te - 1 NA NA - 2 1.624546e-03 3.500275e-04 - 3 NA NA - 4 NA NA - 5 1.273993e-04 8.827508e-04 - 6 NA NA - 7 NA NA - 8 NA NA - 9 NA NA - 10 3.479209e-04 4.446702e-05 - 11 NA NA - 12 1.399986e-07 5.469797e-04 - 13 NA NA - 14 NA NA - 15 1.334537e-03 8.410892e-04 - 16 NA NA - 17 NA NA - 18 NA NA - 19 NA NA - 20 NA NA + err sq_err err_te sq_err_te + 1 NA NA NA NA + 2 0.0403056552 1.624546e-03 0.01870902 3.500275e-04 + 3 NA NA NA NA + 4 NA NA NA NA + 5 0.0112871320 1.273993e-04 -0.02971112 8.827508e-04 + 6 NA NA NA NA + 7 NA NA NA NA + 8 NA NA NA NA + 9 NA NA NA NA + 10 -0.0186526388 3.479209e-04 -0.00666836 4.446702e-05 + 11 NA NA NA NA + 12 0.0003741639 1.399986e-07 0.02338760 5.469797e-04 + 13 NA NA NA NA + 14 NA NA NA NA + 15 -0.0365313171 1.334537e-03 -0.02900154 8.410892e-04 + 16 NA NA NA NA + 17 NA NA NA NA + 18 NA NA NA NA + 19 NA NA NA NA + 20 NA NA NA NA --- @@ -168,27 +168,27 @@ 18 18 700 199 0.2842857 futility C 19 19 1300 345 0.2653846 futility C 20 20 300 86 0.2866667 futility C - sq_err sq_err_te - 1 1.027946e-03 3.315766e-04 - 2 1.624546e-03 3.500275e-04 - 3 4.505182e-04 4.430545e-06 - 4 4.911235e-05 6.079230e-05 - 5 1.273993e-04 8.827508e-04 - 6 2.658014e-04 2.083754e-05 - 7 1.747864e-03 3.881508e-03 - 8 4.469369e-04 8.045543e-04 - 9 5.834430e-04 1.595652e-03 - 10 3.479209e-04 4.446702e-05 - 11 5.576087e-04 9.820124e-04 - 12 1.399986e-07 5.469797e-04 - 13 4.667364e-03 1.574570e-03 - 14 4.289623e-03 1.718314e-02 - 15 1.334537e-03 8.410892e-04 - 16 3.056154e-03 1.747921e-03 - 17 1.085232e-04 1.639552e-03 - 18 1.006312e-03 2.451906e-05 - 19 2.303900e-05 3.488792e-04 - 20 5.731230e-03 5.154508e-03 + err sq_err err_te sq_err_te + 1 -0.0320615980 1.027946e-03 0.018209246 3.315766e-04 + 2 0.0403056552 1.624546e-03 0.018709023 3.500275e-04 + 3 -0.0212254131 4.505182e-04 -0.002104886 4.430545e-06 + 4 -0.0070080201 4.911235e-05 -0.007796942 6.079230e-05 + 5 0.0112871320 1.273993e-04 -0.029711123 8.827508e-04 + 6 -0.0163034180 2.658014e-04 -0.004564815 2.083754e-05 + 7 0.0418074601 1.747864e-03 0.062301746 3.881508e-03 + 8 -0.0211408830 4.469369e-04 -0.028364667 8.045543e-04 + 9 0.0241545653 5.834430e-04 0.039945617 1.595652e-03 + 10 -0.0186526388 3.479209e-04 -0.006668360 4.446702e-05 + 11 0.0236137388 5.576087e-04 0.031337078 9.820124e-04 + 12 0.0003741639 1.399986e-07 0.023387597 5.469797e-04 + 13 0.0683181104 4.667364e-03 0.039680849 1.574570e-03 + 14 0.0654952119 4.289623e-03 0.131084491 1.718314e-02 + 15 -0.0365313171 1.334537e-03 -0.029001538 8.410892e-04 + 16 0.0552824892 3.056154e-03 0.041808151 1.747921e-03 + 17 0.0104174470 1.085232e-04 0.040491379 1.639552e-03 + 18 0.0317224271 1.006312e-03 -0.004951673 2.451906e-05 + 19 -0.0047998954 2.303900e-05 -0.018678309 3.488792e-04 + 20 0.0757048898 5.731230e-03 0.071794904 5.154508e-03 --- @@ -216,27 +216,27 @@ 18 18 700 199 0.2842857 futility C 19 19 1300 345 0.2653846 futility C 20 20 300 86 0.2866667 futility C - sq_err sq_err_te - 1 1.027946e-03 3.315766e-04 - 2 1.624546e-03 3.500275e-04 - 3 4.505182e-04 4.430545e-06 - 4 4.911235e-05 6.079230e-05 - 5 1.273993e-04 8.827508e-04 - 6 2.658014e-04 2.083754e-05 - 7 1.747864e-03 3.881508e-03 - 8 4.469369e-04 8.045543e-04 - 9 5.834430e-04 1.595652e-03 - 10 3.479209e-04 4.446702e-05 - 11 5.576087e-04 9.820124e-04 - 12 1.399986e-07 5.469797e-04 - 13 4.667364e-03 1.574570e-03 - 14 4.289623e-03 1.718314e-02 - 15 1.334537e-03 8.410892e-04 - 16 3.056154e-03 1.747921e-03 - 17 1.085232e-04 1.639552e-03 - 18 1.006312e-03 2.451906e-05 - 19 2.303900e-05 3.488792e-04 - 20 5.731230e-03 5.154508e-03 + err sq_err err_te sq_err_te + 1 -0.0320615980 1.027946e-03 0.018209246 3.315766e-04 + 2 0.0403056552 1.624546e-03 0.018709023 3.500275e-04 + 3 -0.0212254131 4.505182e-04 -0.002104886 4.430545e-06 + 4 -0.0070080201 4.911235e-05 -0.007796942 6.079230e-05 + 5 0.0112871320 1.273993e-04 -0.029711123 8.827508e-04 + 6 -0.0163034180 2.658014e-04 -0.004564815 2.083754e-05 + 7 0.0418074601 1.747864e-03 0.062301746 3.881508e-03 + 8 -0.0211408830 4.469369e-04 -0.028364667 8.045543e-04 + 9 0.0241545653 5.834430e-04 0.039945617 1.595652e-03 + 10 -0.0186526388 3.479209e-04 -0.006668360 4.446702e-05 + 11 0.0236137388 5.576087e-04 0.031337078 9.820124e-04 + 12 0.0003741639 1.399986e-07 0.023387597 5.469797e-04 + 13 0.0683181104 4.667364e-03 0.039680849 1.574570e-03 + 14 0.0654952119 4.289623e-03 0.131084491 1.718314e-02 + 15 -0.0365313171 1.334537e-03 -0.029001538 8.410892e-04 + 16 0.0552824892 3.056154e-03 0.041808151 1.747921e-03 + 17 0.0104174470 1.085232e-04 0.040491379 1.639552e-03 + 18 0.0317224271 1.006312e-03 -0.004951673 2.451906e-05 + 19 -0.0047998954 2.303900e-05 -0.018678309 3.488792e-04 + 20 0.0757048898 5.731230e-03 0.071794904 5.154508e-03 --- @@ -264,27 +264,27 @@ 18 18 700 199 0.2842857 futility C 19 19 1300 345 0.2653846 futility C 20 20 300 86 0.2866667 futility C - sq_err sq_err_te - 1 1.027946e-03 3.315766e-04 - 2 1.624546e-03 3.500275e-04 - 3 4.505182e-04 4.430545e-06 - 4 4.911235e-05 6.079230e-05 - 5 1.273993e-04 8.827508e-04 - 6 2.658014e-04 2.083754e-05 - 7 1.747864e-03 3.881508e-03 - 8 4.469369e-04 8.045543e-04 - 9 5.834430e-04 1.595652e-03 - 10 3.479209e-04 4.446702e-05 - 11 5.576087e-04 9.820124e-04 - 12 1.399986e-07 5.469797e-04 - 13 4.667364e-03 1.574570e-03 - 14 4.289623e-03 1.718314e-02 - 15 1.334537e-03 8.410892e-04 - 16 3.056154e-03 1.747921e-03 - 17 1.085232e-04 1.639552e-03 - 18 1.006312e-03 2.451906e-05 - 19 2.303900e-05 3.488792e-04 - 20 5.731230e-03 5.154508e-03 + err sq_err err_te sq_err_te + 1 -0.0320615980 1.027946e-03 0.018209246 3.315766e-04 + 2 0.0403056552 1.624546e-03 0.018709023 3.500275e-04 + 3 -0.0212254131 4.505182e-04 -0.002104886 4.430545e-06 + 4 -0.0070080201 4.911235e-05 -0.007796942 6.079230e-05 + 5 0.0112871320 1.273993e-04 -0.029711123 8.827508e-04 + 6 -0.0163034180 2.658014e-04 -0.004564815 2.083754e-05 + 7 0.0418074601 1.747864e-03 0.062301746 3.881508e-03 + 8 -0.0211408830 4.469369e-04 -0.028364667 8.045543e-04 + 9 0.0241545653 5.834430e-04 0.039945617 1.595652e-03 + 10 -0.0186526388 3.479209e-04 -0.006668360 4.446702e-05 + 11 0.0236137388 5.576087e-04 0.031337078 9.820124e-04 + 12 0.0003741639 1.399986e-07 0.023387597 5.469797e-04 + 13 0.0683181104 4.667364e-03 0.039680849 1.574570e-03 + 14 0.0654952119 4.289623e-03 0.131084491 1.718314e-02 + 15 -0.0365313171 1.334537e-03 -0.029001538 8.410892e-04 + 16 0.0552824892 3.056154e-03 0.041808151 1.747921e-03 + 17 0.0104174470 1.085232e-04 0.040491379 1.639552e-03 + 18 0.0317224271 1.006312e-03 -0.004951673 2.451906e-05 + 19 -0.0047998954 2.303900e-05 -0.018678309 3.488792e-04 + 20 0.0757048898 5.731230e-03 0.071794904 5.154508e-03 --- @@ -312,27 +312,27 @@ 18 18 700 199 0.2842857 futility C 19 19 1300 345 0.2653846 futility C 20 20 300 86 0.2866667 futility C - sq_err sq_err_te - 1 1.027946e-03 3.315766e-04 - 2 1.624546e-03 3.500275e-04 - 3 4.505182e-04 4.430545e-06 - 4 4.911235e-05 6.079230e-05 - 5 1.273993e-04 8.827508e-04 - 6 2.658014e-04 2.083754e-05 - 7 1.747864e-03 3.881508e-03 - 8 4.469369e-04 8.045543e-04 - 9 5.834430e-04 1.595652e-03 - 10 3.479209e-04 4.446702e-05 - 11 5.576087e-04 9.820124e-04 - 12 1.399986e-07 5.469797e-04 - 13 4.667364e-03 1.574570e-03 - 14 4.289623e-03 1.718314e-02 - 15 1.334537e-03 8.410892e-04 - 16 3.056154e-03 1.747921e-03 - 17 1.085232e-04 1.639552e-03 - 18 1.006312e-03 2.451906e-05 - 19 2.303900e-05 3.488792e-04 - 20 5.731230e-03 5.154508e-03 + err sq_err err_te sq_err_te + 1 -0.0320615980 1.027946e-03 0.018209246 3.315766e-04 + 2 0.0403056552 1.624546e-03 0.018709023 3.500275e-04 + 3 -0.0212254131 4.505182e-04 -0.002104886 4.430545e-06 + 4 -0.0070080201 4.911235e-05 -0.007796942 6.079230e-05 + 5 0.0112871320 1.273993e-04 -0.029711123 8.827508e-04 + 6 -0.0163034180 2.658014e-04 -0.004564815 2.083754e-05 + 7 0.0418074601 1.747864e-03 0.062301746 3.881508e-03 + 8 -0.0211408830 4.469369e-04 -0.028364667 8.045543e-04 + 9 0.0241545653 5.834430e-04 0.039945617 1.595652e-03 + 10 -0.0186526388 3.479209e-04 -0.006668360 4.446702e-05 + 11 0.0236137388 5.576087e-04 0.031337078 9.820124e-04 + 12 0.0003741639 1.399986e-07 0.023387597 5.469797e-04 + 13 0.0683181104 4.667364e-03 0.039680849 1.574570e-03 + 14 0.0654952119 4.289623e-03 0.131084491 1.718314e-02 + 15 -0.0365313171 1.334537e-03 -0.029001538 8.410892e-04 + 16 0.0552824892 3.056154e-03 0.041808151 1.747921e-03 + 17 0.0104174470 1.085232e-04 0.040491379 1.639552e-03 + 18 0.0317224271 1.006312e-03 -0.004951673 2.451906e-05 + 19 -0.0047998954 2.303900e-05 -0.018678309 3.488792e-04 + 20 0.0757048898 5.731230e-03 0.071794904 5.154508e-03 --- @@ -360,27 +360,27 @@ 18 18 700 199 0.2842857 futility 19 19 1300 345 0.2653846 futility 20 20 300 86 0.2866667 futility - sq_err sq_err_te - 1 NA NA - 2 1.624546e-03 3.500275e-04 - 3 NA NA - 4 NA NA - 5 1.273993e-04 8.827508e-04 - 6 NA NA - 7 NA NA - 8 NA NA - 9 NA NA - 10 3.479209e-04 4.446702e-05 - 11 NA NA - 12 1.399986e-07 5.469797e-04 - 13 NA NA - 14 NA NA - 15 1.334537e-03 8.410892e-04 - 16 NA NA - 17 NA NA - 18 NA NA - 19 NA NA - 20 NA NA + err sq_err err_te sq_err_te + 1 NA NA NA NA + 2 0.0403056552 1.624546e-03 0.01870902 3.500275e-04 + 3 NA NA NA NA + 4 NA NA NA NA + 5 0.0112871320 1.273993e-04 -0.02971112 8.827508e-04 + 6 NA NA NA NA + 7 NA NA NA NA + 8 NA NA NA NA + 9 NA NA NA NA + 10 -0.0186526388 3.479209e-04 -0.00666836 4.446702e-05 + 11 NA NA NA NA + 12 0.0003741639 1.399986e-07 0.02338760 5.469797e-04 + 13 NA NA NA NA + 14 NA NA NA NA + 15 -0.0365313171 1.334537e-03 -0.02900154 8.410892e-04 + 16 NA NA NA NA + 17 NA NA NA NA + 18 NA NA NA NA + 19 NA NA NA NA + 20 NA NA NA NA --- @@ -408,27 +408,27 @@ 18 18 700 199 0.2842857 futility 19 19 1300 345 0.2653846 futility 20 20 300 86 0.2866667 futility - sq_err sq_err_te - 1 NA NA - 2 1.624546e-03 NA - 3 NA NA - 4 NA NA - 5 1.273993e-04 NA - 6 NA NA - 7 NA NA - 8 NA NA - 9 NA NA - 10 3.479209e-04 NA - 11 NA NA - 12 1.399986e-07 NA - 13 NA NA - 14 NA NA - 15 1.334537e-03 NA - 16 NA NA - 17 NA NA - 18 NA NA - 19 NA NA - 20 NA NA + err sq_err err_te sq_err_te + 1 NA NA NA NA + 2 0.0403056552 1.624546e-03 NA NA + 3 NA NA NA NA + 4 NA NA NA NA + 5 0.0112871320 1.273993e-04 NA NA + 6 NA NA NA NA + 7 NA NA NA NA + 8 NA NA NA NA + 9 NA NA NA NA + 10 -0.0186526388 3.479209e-04 NA NA + 11 NA NA NA NA + 12 0.0003741639 1.399986e-07 NA NA + 13 NA NA NA NA + 14 NA NA NA NA + 15 -0.0365313171 1.334537e-03 NA NA + 16 NA NA NA NA + 17 NA NA NA NA + 18 NA NA NA NA + 19 NA NA NA NA + 20 NA NA NA NA --- @@ -456,27 +456,27 @@ 18 18 700 199 0.2842857 futility 19 19 1300 345 0.2653846 futility 20 20 300 86 0.2866667 futility - sq_err sq_err_te - 1 NA NA - 2 0.0015699537 4.466986e-04 - 3 NA NA - 4 NA NA - 5 0.0001204640 8.265865e-04 - 6 NA NA - 7 NA NA - 8 NA NA - 9 NA NA - 10 0.0003818206 1.809316e-05 - 11 NA NA - 12 0.0000000000 6.634527e-04 - 13 NA NA - 14 NA NA - 15 0.0013741082 8.004110e-04 - 16 NA NA - 17 NA NA - 18 NA NA - 19 NA NA - 20 NA NA + err sq_err err_te sq_err_te + 1 NA NA NA NA + 2 0.03962264 0.0015699537 0.021135247 4.466986e-04 + 3 NA NA NA NA + 4 NA NA NA NA + 5 0.01097561 0.0001204640 -0.028750418 8.265865e-04 + 6 NA NA NA NA + 7 NA NA NA NA + 8 NA NA NA NA + 9 NA NA NA NA + 10 -0.01954023 0.0003818206 -0.004253606 1.809316e-05 + 11 NA NA NA NA + 12 0.00000000 0.0000000000 0.025757576 6.634527e-04 + 13 NA NA NA NA + 14 NA NA NA NA + 15 -0.03706897 0.0013741082 -0.028291536 8.004110e-04 + 16 NA NA NA NA + 17 NA NA NA NA + 18 NA NA NA NA + 19 NA NA NA NA + 20 NA NA NA NA # Tidy results of simulations from binomial-outcome trial without common control works @@ -504,27 +504,27 @@ 18 18 2000 472 0.2360000 max 19 19 1600 372 0.2325000 superiority B B 20 20 500 135 0.2700000 superiority B B - sq_err sq_err_te - 1 1.403445e-03 NA - 2 6.546625e-05 NA - 3 8.482272e-05 NA - 4 6.368035e-04 NA - 5 1.355680e-05 NA - 6 1.071614e-03 NA - 7 1.475723e-06 NA - 8 NA NA - 9 1.059480e-05 NA - 10 NA NA - 11 1.336775e-04 NA - 12 1.254923e-03 NA - 13 NA NA - 14 NA NA - 15 3.127227e-04 NA - 16 NA NA - 17 NA NA - 18 NA NA - 19 2.154692e-05 NA - 20 4.407718e-05 NA + err sq_err err_te sq_err_te + 1 -0.037462579 1.403445e-03 NA NA + 2 -0.008091122 6.546625e-05 NA NA + 3 -0.009209925 8.482272e-05 NA NA + 4 -0.025234966 6.368035e-04 NA NA + 5 -0.003681956 1.355680e-05 NA NA + 6 -0.032735518 1.071614e-03 NA NA + 7 -0.001214794 1.475723e-06 NA NA + 8 NA NA NA NA + 9 -0.003254965 1.059480e-05 NA NA + 10 NA NA NA NA + 11 -0.011561899 1.336775e-04 NA NA + 12 -0.035424899 1.254923e-03 NA NA + 13 NA NA NA NA + 14 NA NA NA NA + 15 -0.017683969 3.127227e-04 NA NA + 16 NA NA NA NA + 17 NA NA NA NA + 18 NA NA NA NA + 19 0.004641866 2.154692e-05 NA NA + 20 0.006639065 4.407718e-05 NA NA --- @@ -552,27 +552,27 @@ 18 18 2000 472 0.2360000 max 19 19 1600 372 0.2325000 superiority B B 20 20 500 135 0.2700000 superiority B B - sq_err sq_err_te - 1 1.403445e-03 NA - 2 6.546625e-05 NA - 3 8.482272e-05 NA - 4 6.368035e-04 NA - 5 1.355680e-05 NA - 6 1.071614e-03 NA - 7 1.475723e-06 NA - 8 NA NA - 9 1.059480e-05 NA - 10 NA NA - 11 1.336775e-04 NA - 12 1.254923e-03 NA - 13 NA NA - 14 NA NA - 15 3.127227e-04 NA - 16 NA NA - 17 NA NA - 18 NA NA - 19 2.154692e-05 NA - 20 4.407718e-05 NA + err sq_err err_te sq_err_te + 1 -0.037462579 1.403445e-03 NA NA + 2 -0.008091122 6.546625e-05 NA NA + 3 -0.009209925 8.482272e-05 NA NA + 4 -0.025234966 6.368035e-04 NA NA + 5 -0.003681956 1.355680e-05 NA NA + 6 -0.032735518 1.071614e-03 NA NA + 7 -0.001214794 1.475723e-06 NA NA + 8 NA NA NA NA + 9 -0.003254965 1.059480e-05 NA NA + 10 NA NA NA NA + 11 -0.011561899 1.336775e-04 NA NA + 12 -0.035424899 1.254923e-03 NA NA + 13 NA NA NA NA + 14 NA NA NA NA + 15 -0.017683969 3.127227e-04 NA NA + 16 NA NA NA NA + 17 NA NA NA NA + 18 NA NA NA NA + 19 0.004641866 2.154692e-05 NA NA + 20 0.006639065 4.407718e-05 NA NA --- @@ -600,27 +600,27 @@ 18 18 2000 472 0.2360000 max B 19 19 1600 372 0.2325000 superiority B B 20 20 500 135 0.2700000 superiority B B - sq_err sq_err_te - 1 1.403445e-03 NA - 2 6.546625e-05 NA - 3 8.482272e-05 NA - 4 6.368035e-04 NA - 5 1.355680e-05 NA - 6 1.071614e-03 NA - 7 1.475723e-06 NA - 8 1.840904e-09 NA - 9 1.059480e-05 NA - 10 4.775573e-10 NA - 11 1.336775e-04 NA - 12 1.254923e-03 NA - 13 2.426079e-04 NA - 14 1.405516e-08 NA - 15 3.127227e-04 NA - 16 1.942916e-04 NA - 17 2.169804e-03 NA - 18 3.551297e-04 NA - 19 2.154692e-05 NA - 20 4.407718e-05 NA + err sq_err err_te sq_err_te + 1 -3.746258e-02 1.403445e-03 NA NA + 2 -8.091122e-03 6.546625e-05 NA NA + 3 -9.209925e-03 8.482272e-05 NA NA + 4 -2.523497e-02 6.368035e-04 NA NA + 5 -3.681956e-03 1.355680e-05 NA NA + 6 -3.273552e-02 1.071614e-03 NA NA + 7 -1.214794e-03 1.475723e-06 NA NA + 8 4.290576e-05 1.840904e-09 NA NA + 9 -3.254965e-03 1.059480e-05 NA NA + 10 2.185308e-05 4.775573e-10 NA NA + 11 -1.156190e-02 1.336775e-04 NA NA + 12 -3.542490e-02 1.254923e-03 NA NA + 13 1.557587e-02 2.426079e-04 NA NA + 14 1.185545e-04 1.405516e-08 NA NA + 15 -1.768397e-02 3.127227e-04 NA NA + 16 1.393885e-02 1.942916e-04 NA NA + 17 -4.658115e-02 2.169804e-03 NA NA + 18 1.884488e-02 3.551297e-04 NA NA + 19 4.641866e-03 2.154692e-05 NA NA + 20 6.639065e-03 4.407718e-05 NA NA --- @@ -648,27 +648,27 @@ 18 18 2000 472 0.2360000 max B 19 19 1600 372 0.2325000 superiority B B 20 20 500 135 0.2700000 superiority B B - sq_err sq_err_te - 1 1.403445e-03 NA - 2 6.546625e-05 NA - 3 8.482272e-05 NA - 4 6.368035e-04 NA - 5 1.355680e-05 NA - 6 1.071614e-03 NA - 7 1.475723e-06 NA - 8 1.840904e-09 NA - 9 1.059480e-05 NA - 10 4.775573e-10 NA - 11 1.336775e-04 NA - 12 1.254923e-03 NA - 13 2.426079e-04 NA - 14 1.405516e-08 NA - 15 3.127227e-04 NA - 16 1.942916e-04 NA - 17 1.157961e-04 NA - 18 3.551297e-04 NA - 19 2.154692e-05 NA - 20 4.407718e-05 NA + err sq_err err_te sq_err_te + 1 -3.746258e-02 1.403445e-03 NA NA + 2 -8.091122e-03 6.546625e-05 NA NA + 3 -9.209925e-03 8.482272e-05 NA NA + 4 -2.523497e-02 6.368035e-04 NA NA + 5 -3.681956e-03 1.355680e-05 NA NA + 6 -3.273552e-02 1.071614e-03 NA NA + 7 -1.214794e-03 1.475723e-06 NA NA + 8 4.290576e-05 1.840904e-09 NA NA + 9 -3.254965e-03 1.059480e-05 NA NA + 10 2.185308e-05 4.775573e-10 NA NA + 11 -1.156190e-02 1.336775e-04 NA NA + 12 -3.542490e-02 1.254923e-03 NA NA + 13 1.557587e-02 2.426079e-04 NA NA + 14 1.185545e-04 1.405516e-08 NA NA + 15 -1.768397e-02 3.127227e-04 NA NA + 16 1.393885e-02 1.942916e-04 NA NA + 17 1.076086e-02 1.157961e-04 NA NA + 18 1.884488e-02 3.551297e-04 NA NA + 19 4.641866e-03 2.154692e-05 NA NA + 20 6.639065e-03 4.407718e-05 NA NA --- @@ -696,27 +696,27 @@ 18 18 2000 472 0.2360000 max B 19 19 1600 372 0.2325000 superiority B B 20 20 500 135 0.2700000 superiority B B - sq_err sq_err_te - 1 1.403445e-03 NA - 2 6.546625e-05 NA - 3 8.482272e-05 NA - 4 6.368035e-04 NA - 5 1.355680e-05 NA - 6 1.071614e-03 NA - 7 1.475723e-06 NA - 8 1.840904e-09 NA - 9 1.059480e-05 NA - 10 4.775573e-10 NA - 11 1.336775e-04 NA - 12 1.254923e-03 NA - 13 2.426079e-04 NA - 14 1.405516e-08 NA - 15 3.127227e-04 NA - 16 1.942916e-04 NA - 17 1.157961e-04 NA - 18 3.551297e-04 NA - 19 2.154692e-05 NA - 20 4.407718e-05 NA + err sq_err err_te sq_err_te + 1 -3.746258e-02 1.403445e-03 NA NA + 2 -8.091122e-03 6.546625e-05 NA NA + 3 -9.209925e-03 8.482272e-05 NA NA + 4 -2.523497e-02 6.368035e-04 NA NA + 5 -3.681956e-03 1.355680e-05 NA NA + 6 -3.273552e-02 1.071614e-03 NA NA + 7 -1.214794e-03 1.475723e-06 NA NA + 8 4.290576e-05 1.840904e-09 NA NA + 9 -3.254965e-03 1.059480e-05 NA NA + 10 2.185308e-05 4.775573e-10 NA NA + 11 -1.156190e-02 1.336775e-04 NA NA + 12 -3.542490e-02 1.254923e-03 NA NA + 13 1.557587e-02 2.426079e-04 NA NA + 14 1.185545e-04 1.405516e-08 NA NA + 15 -1.768397e-02 3.127227e-04 NA NA + 16 1.393885e-02 1.942916e-04 NA NA + 17 1.076086e-02 1.157961e-04 NA NA + 18 1.884488e-02 3.551297e-04 NA NA + 19 4.641866e-03 2.154692e-05 NA NA + 20 6.639065e-03 4.407718e-05 NA NA --- @@ -744,27 +744,27 @@ 18 18 2000 472 0.2360000 max 19 19 1600 372 0.2325000 superiority B B 20 20 500 135 0.2700000 superiority B B - sq_err sq_err_te - 1 1.403445e-03 9.599721e-04 - 2 6.546625e-05 4.826991e-03 - 3 8.482272e-05 7.837008e-04 - 4 6.368035e-04 1.392739e-08 - 5 1.355680e-05 8.206064e-04 - 6 1.071614e-03 5.625415e-05 - 7 1.475723e-06 1.806622e-03 - 8 NA NA - 9 1.059480e-05 6.094579e-04 - 10 NA NA - 11 1.336775e-04 1.055027e-03 - 12 1.254923e-03 5.788501e-04 - 13 NA NA - 14 NA NA - 15 3.127227e-04 4.475481e-04 - 16 NA NA - 17 NA NA - 18 NA NA - 19 2.154692e-05 5.878924e-04 - 20 4.407718e-05 4.868020e-03 + err sq_err err_te sq_err_te + 1 -0.037462579 1.403445e-03 -0.0309834173 9.599721e-04 + 2 -0.008091122 6.546625e-05 -0.0694765496 4.826991e-03 + 3 -0.009209925 8.482272e-05 0.0279946570 7.837008e-04 + 4 -0.025234966 6.368035e-04 0.0001180144 1.392739e-08 + 5 -0.003681956 1.355680e-05 -0.0286462280 8.206064e-04 + 6 -0.032735518 1.071614e-03 0.0075002766 5.625415e-05 + 7 -0.001214794 1.475723e-06 -0.0425043761 1.806622e-03 + 8 NA NA NA NA + 9 -0.003254965 1.059480e-05 0.0246872011 6.094579e-04 + 10 NA NA NA NA + 11 -0.011561899 1.336775e-04 0.0324811838 1.055027e-03 + 12 -0.035424899 1.254923e-03 -0.0240593040 5.788501e-04 + 13 NA NA NA NA + 14 NA NA NA NA + 15 -0.017683969 3.127227e-04 0.0211553332 4.475481e-04 + 16 NA NA NA NA + 17 NA NA NA NA + 18 NA NA NA NA + 19 0.004641866 2.154692e-05 0.0242464916 5.878924e-04 + 20 0.006639065 4.407718e-05 -0.0697711964 4.868020e-03 --- @@ -792,27 +792,75 @@ 18 18 2000 472 0.2360000 max 19 19 1600 372 0.2325000 superiority B B 20 20 500 135 0.2700000 superiority B B - sq_err sq_err_te - 1 1.537870e-03 NA - 2 8.264463e-05 NA - 3 8.944582e-05 NA - 4 6.941805e-04 NA - 5 1.712247e-05 NA - 6 1.185913e-03 NA - 7 1.484945e-06 NA - 8 NA NA - 9 1.371742e-05 NA - 10 NA NA - 11 1.397028e-04 NA - 12 1.455445e-03 NA - 13 NA NA - 14 NA NA - 15 3.354009e-04 NA - 16 NA NA - 17 NA NA - 18 NA NA - 19 1.932134e-05 NA - 20 2.146659e-05 NA + err sq_err err_te sq_err_te + 1 -0.039215686 1.537870e-03 NA NA + 2 -0.009090909 8.264463e-05 NA NA + 3 -0.009457580 8.944582e-05 NA NA + 4 -0.026347305 6.941805e-04 NA NA + 5 -0.004137931 1.712247e-05 NA NA + 6 -0.034437086 1.185913e-03 NA NA + 7 -0.001218583 1.484945e-06 NA NA + 8 NA NA NA NA + 9 -0.003703704 1.371742e-05 NA NA + 10 NA NA NA NA + 11 -0.011819596 1.397028e-04 NA NA + 12 -0.038150289 1.455445e-03 NA NA + 13 NA NA NA NA + 14 NA NA NA NA + 15 -0.018313953 3.354009e-04 NA NA + 16 NA NA NA NA + 17 NA NA NA NA + 18 NA NA NA NA + 19 0.004395604 1.932134e-05 NA NA + 20 0.004633205 2.146659e-05 NA NA + +# Selection works with practical equivalence and outcome-data lag works + + Code + extract_results(res, select_strategy = "best") + Output + sim final_n sum_ys ratio_ys final_status superior_arm selected_arm + 1 1 1500 351 0.2340000 equivalence A + 2 2 1500 339 0.2260000 equivalence B + 3 3 2100 481 0.2290476 max B + 4 4 1500 349 0.2326667 superiority B B + 5 5 1900 462 0.2431579 equivalence B + 6 6 2100 487 0.2319048 max B + 7 7 1500 344 0.2293333 equivalence B + 8 8 1700 430 0.2529412 equivalence B + 9 9 2100 526 0.2504762 max B + 10 10 200 40 0.2000000 superiority B B + 11 11 1600 377 0.2356250 equivalence A + 12 12 1600 369 0.2306250 equivalence B + 13 13 1800 476 0.2644444 equivalence B + 14 14 2100 508 0.2419048 equivalence B + 15 15 200 51 0.2550000 superiority B B + 16 16 2100 480 0.2285714 max B + 17 17 2100 482 0.2295238 max B + 18 18 1500 339 0.2260000 equivalence B + 19 19 1500 352 0.2346667 equivalence A + 20 20 1900 440 0.2315789 equivalence A + err sq_err err_te sq_err_te + 1 -0.020294048 4.118484e-04 NA NA + 2 -0.007014838 4.920796e-05 NA NA + 3 -0.014463951 2.092059e-04 NA NA + 4 -0.024881085 6.190684e-04 NA NA + 5 0.007033408 4.946882e-05 NA NA + 6 -0.017663275 3.119913e-04 NA NA + 7 -0.003116444 9.712220e-06 NA NA + 8 0.017648868 3.114825e-04 NA NA + 9 0.015377199 2.364583e-04 NA NA + 10 -0.099895742 9.979159e-03 NA NA + 11 -0.016594114 2.753646e-04 NA NA + 12 -0.001521558 2.315138e-06 NA NA + 13 0.031967991 1.021952e-03 NA NA + 14 0.007552201 5.703574e-05 NA NA + 15 -0.047414042 2.248091e-03 NA NA + 16 -0.014629099 2.140105e-04 NA NA + 17 -0.006069625 3.684034e-05 NA NA + 18 -0.006869535 4.719051e-05 NA NA + 19 -0.014451375 2.088422e-04 NA NA + 20 -0.020764240 4.311537e-04 NA NA # Metric history of specific trial works diff --git a/tests/testthat/_snaps/plot_convergence/convergence-plot-binomial-size-mean-rmse-selected.svg b/tests/testthat/_snaps/plot_convergence/convergence-plot-binomial-size-mean-rmse-selected.svg index 5431b71..fbf8191 100644 --- a/tests/testthat/_snaps/plot_convergence/convergence-plot-binomial-size-mean-rmse-selected.svg +++ b/tests/testthat/_snaps/plot_convergence/convergence-plot-binomial-size-mean-rmse-selected.svg @@ -18,7 +18,7 @@ - + diff --git a/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-pct-followed-n-2-cores.svg b/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-pct-followed-n-2-cores.svg index 2ea4574..8648422 100644 --- a/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-pct-followed-n-2-cores.svg +++ b/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-pct-followed-n-2-cores.svg @@ -80,15 +80,15 @@ 1500 Total no. of patients with outcome data Percentage with outcome data - - - - - - -A -B -C + + + + + + +A +B +C history plot, binomial, multiple, pct, followed n, 2 cores diff --git a/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-pct-followed-n.svg b/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-pct-followed-n.svg index 415215f..3b1f4c7 100644 --- a/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-pct-followed-n.svg +++ b/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-pct-followed-n.svg @@ -80,15 +80,15 @@ 1500 Total no. of patients with outcome data Percentage with outcome data - - - - - - -A -B -C + + + + + + +A +B +C history plot, binomial, multiple, pct, followed n diff --git a/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-pct-total-n.svg b/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-pct-total-n.svg index 406c271..ac9fddd 100644 --- a/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-pct-total-n.svg +++ b/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-pct-total-n.svg @@ -80,15 +80,15 @@ 1500 Total no. of patients randomised Percentage with outcome data - - - - - - -A -B -C + + + + + + +A +B +C history plot, binomial, multiple, pct, total n diff --git a/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-pct.svg b/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-pct.svg index 174af11..6016013 100644 --- a/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-pct.svg +++ b/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-pct.svg @@ -80,15 +80,15 @@ 16 Look Percentage with outcome data - - - - - - -A -B -C + + + + + + +A +B +C history plot, binomial, multiple, pct diff --git a/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-prob.svg b/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-prob.svg index 9cb17b2..ad2da8c 100644 --- a/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-prob.svg +++ b/tests/testthat/_snaps/plot_history/history-plot-binomial-multiple-prob.svg @@ -80,15 +80,15 @@ 16 Look Allocation probability - - - - - - -A -B -C + + + + + + +A +B +C history plot, binomial, multiple, prob diff --git a/tests/testthat/_snaps/plot_history/history-plot-binomial-single-n-look.svg b/tests/testthat/_snaps/plot_history/history-plot-binomial-single-n-look.svg index 270982c..319ff12 100644 --- a/tests/testthat/_snaps/plot_history/history-plot-binomial-single-n-look.svg +++ b/tests/testthat/_snaps/plot_history/history-plot-binomial-single-n-look.svg @@ -66,12 +66,12 @@ 7 Look No. of patients with outcome data - - - -A -B -C + + + +A +B +C history plot, binomial, single, n, look diff --git a/tests/testthat/_snaps/plot_history/history-plot-binomial-single-pct-all-look.svg b/tests/testthat/_snaps/plot_history/history-plot-binomial-single-pct-all-look.svg index ccf5a7c..989660e 100644 --- a/tests/testthat/_snaps/plot_history/history-plot-binomial-single-pct-all-look.svg +++ b/tests/testthat/_snaps/plot_history/history-plot-binomial-single-pct-all-look.svg @@ -79,12 +79,12 @@ 7 Look Percentage randomised - - - -A -B -C + + + +A +B +C history plot, binomial, single, pct all, look diff --git a/tests/testthat/_snaps/plot_history/history-plot-binomial-single-pct-look.svg b/tests/testthat/_snaps/plot_history/history-plot-binomial-single-pct-look.svg index e4f6e54..e5e0992 100644 --- a/tests/testthat/_snaps/plot_history/history-plot-binomial-single-pct-look.svg +++ b/tests/testthat/_snaps/plot_history/history-plot-binomial-single-pct-look.svg @@ -79,12 +79,12 @@ 7 Look Percentage with outcome data - - - -A -B -C + + + +A +B +C history plot, binomial, single, pct, look diff --git a/tests/testthat/_snaps/plot_history/history-plot-binomial-single-prob-followed-n.svg b/tests/testthat/_snaps/plot_history/history-plot-binomial-single-prob-followed-n.svg index 83cbadb..e954a5d 100644 --- a/tests/testthat/_snaps/plot_history/history-plot-binomial-single-prob-followed-n.svg +++ b/tests/testthat/_snaps/plot_history/history-plot-binomial-single-prob-followed-n.svg @@ -71,12 +71,12 @@ 750 Total no. of patients with outcome data Allocation probability - - - -A -B -C + + + +A +B +C history plot, binomial, single, prob, followed n diff --git a/tests/testthat/_snaps/plot_history/history-plot-binomial-single-prob-total-n.svg b/tests/testthat/_snaps/plot_history/history-plot-binomial-single-prob-total-n.svg index 4a0677f..a9bd853 100644 --- a/tests/testthat/_snaps/plot_history/history-plot-binomial-single-prob-total-n.svg +++ b/tests/testthat/_snaps/plot_history/history-plot-binomial-single-prob-total-n.svg @@ -71,12 +71,12 @@ 750 Total no. of patients randomised Allocation probability - - - -A -B -C + + + +A +B +C history plot, binomial, single, prob, total n diff --git a/tests/testthat/_snaps/plot_history/history-plot-binomial-single-prob.svg b/tests/testthat/_snaps/plot_history/history-plot-binomial-single-prob.svg index c0f50fd..f42f23b 100644 --- a/tests/testthat/_snaps/plot_history/history-plot-binomial-single-prob.svg +++ b/tests/testthat/_snaps/plot_history/history-plot-binomial-single-prob.svg @@ -79,12 +79,12 @@ 7 Look Allocation probability - - - -A -B -C + + + +A +B +C history plot, binomial, single, prob diff --git a/tests/testthat/_snaps/plot_history/history-plot-binomial-single-ratio-ys-all-look.svg b/tests/testthat/_snaps/plot_history/history-plot-binomial-single-ratio-ys-all-look.svg index 2ed3325..f1dff33 100644 --- a/tests/testthat/_snaps/plot_history/history-plot-binomial-single-ratio-ys-all-look.svg +++ b/tests/testthat/_snaps/plot_history/history-plot-binomial-single-ratio-ys-all-look.svg @@ -61,12 +61,12 @@ 7 Look Ratio of all outcomes - - - -A -B -C + + + +A +B +C history plot, binomial, single, ratio ys all, look diff --git a/tests/testthat/_snaps/plot_history/history-plot-binomial-single-ratio-ys-look.svg b/tests/testthat/_snaps/plot_history/history-plot-binomial-single-ratio-ys-look.svg index 8a8972e..68badde 100644 --- a/tests/testthat/_snaps/plot_history/history-plot-binomial-single-ratio-ys-look.svg +++ b/tests/testthat/_snaps/plot_history/history-plot-binomial-single-ratio-ys-look.svg @@ -61,12 +61,12 @@ 7 Look Ratio of available outcomes - - - -A -B -C + + + +A +B +C history plot, binomial, single, ratio ys, look diff --git a/tests/testthat/_snaps/plot_metrics_ecdf/errors.svg b/tests/testthat/_snaps/plot_metrics_ecdf/errors.svg new file mode 100644 index 0000000..73d4bd8 --- /dev/null +++ b/tests/testthat/_snaps/plot_metrics_ecdf/errors.svg @@ -0,0 +1,388 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Error^2 + + + + + + + + + +Error TE^2 + + + + + + + + + +Abs(error TE) + + + + + + + + + +Error + + + + + + + + + +Error TE + + + + + + + + + +Abs(error) + + + + + + +0.00 +0.01 +0.02 +0.03 + + + + +0.02 +0.04 +0.06 +0.08 + + + + +-0.08 +-0.06 +-0.04 +-0.02 + + + + +0.000 +0.002 +0.004 +0.006 + + + + +-0.03 +-0.02 +-0.01 +0.00 + + + +0e+00 +5e-04 +1e-03 +0% +20% +40% +60% +80% +100% + + + + + + +0% +20% +40% +60% +80% +100% + + + + + + +0% +20% +40% +60% +80% +100% + + + + + + +0% +20% +40% +60% +80% +100% + + + + + + +0% +20% +40% +60% +80% +100% + + + + + + +0% +20% +40% +60% +80% +100% + + + + + + +Value +errors + + diff --git a/tests/testthat/_snaps/plot_metrics_ecdf/no-restriction.svg b/tests/testthat/_snaps/plot_metrics_ecdf/no-restriction.svg index def90d3..4825667 100644 --- a/tests/testthat/_snaps/plot_metrics_ecdf/no-restriction.svg +++ b/tests/testthat/_snaps/plot_metrics_ecdf/no-restriction.svg @@ -18,7 +18,7 @@ - + diff --git a/tests/testthat/_snaps/plot_metrics_ecdf/selected.svg b/tests/testthat/_snaps/plot_metrics_ecdf/selected.svg index 2adca09..b442d0f 100644 --- a/tests/testthat/_snaps/plot_metrics_ecdf/selected.svg +++ b/tests/testthat/_snaps/plot_metrics_ecdf/selected.svg @@ -18,7 +18,7 @@ - + diff --git a/tests/testthat/_snaps/plot_metrics_ecdf/superior.svg b/tests/testthat/_snaps/plot_metrics_ecdf/superior.svg index 2bd8d23..deef7f4 100644 --- a/tests/testthat/_snaps/plot_metrics_ecdf/superior.svg +++ b/tests/testthat/_snaps/plot_metrics_ecdf/superior.svg @@ -18,7 +18,7 @@ - + diff --git a/tests/testthat/_snaps/plot_status/status-plot-across-arms-binomial.svg b/tests/testthat/_snaps/plot_status/status-plot-across-arms-binomial.svg index 97dba09..e493b6b 100644 --- a/tests/testthat/_snaps/plot_status/status-plot-across-arms-binomial.svg +++ b/tests/testthat/_snaps/plot_status/status-plot-across-arms-binomial.svg @@ -76,14 +76,14 @@ 16 Look Status probabilities - - - - -Recruiting -Futility -Equivalence -Superiority + + + + +Recruiting +Futility +Equivalence +Superiority status plot across arms, binomial diff --git a/tests/testthat/_snaps/plot_status/status-plot-for-all-arms-binomial.svg b/tests/testthat/_snaps/plot_status/status-plot-for-all-arms-binomial.svg index b88291f..a49ce8b 100644 --- a/tests/testthat/_snaps/plot_status/status-plot-for-all-arms-binomial.svg +++ b/tests/testthat/_snaps/plot_status/status-plot-for-all-arms-binomial.svg @@ -20,232 +20,220 @@ - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - + +C - - + + - -C + +B - - + + - - - - - - - - - - -A - - - - - - - - - -B + +A 0 -500 -1000 -1500 -0 -500 -1000 -1500 -0 -500 -1000 -1500 -0% -20% -40% -60% -80% +500 +1000 +1500 +0 +500 +1000 +1500 +0 +500 +1000 +1500 +0% +20% +40% +60% +80% 100% +0% +20% +40% +60% +80% +100% 0% -20% -40% -60% -80% -100% +20% +40% +60% +80% +100% Total no. of patients with outcome data Status probabilities - - - - - -Recruiting -Inferiority -Futility -Equivalence -Superiority + + + + + +Recruiting +Inferiority +Futility +Equivalence +Superiority status plot for all arms, binomial diff --git a/tests/testthat/_snaps/plot_status/status-plot-for-arm-c-binom.svg b/tests/testthat/_snaps/plot_status/status-plot-for-arm-c-binom.svg index fe8791c..9bcb9a8 100644 --- a/tests/testthat/_snaps/plot_status/status-plot-for-arm-c-binom.svg +++ b/tests/testthat/_snaps/plot_status/status-plot-for-arm-c-binom.svg @@ -88,16 +88,16 @@ 100% Total no. of patients randomised Status probabilities - - - - - -Recruiting -Inferiority -Futility -Equivalence -Superiority + + + + + +Recruiting +Inferiority +Futility +Equivalence +Superiority status plot for arm C, binom diff --git a/tests/testthat/_snaps/run_trials.md b/tests/testthat/_snaps/run_trials.md index 9b637d3..2118a13 100644 --- a/tests/testthat/_snaps/run_trials.md +++ b/tests/testthat/_snaps/run_trials.md @@ -80,6 +80,47 @@ * Number of posterior draws: 5000 * Posterior estimation method: medians with MAD-SDs +--- + + Code + run_trial(setup_rescale_probs, seed = 12345) + Output + Single simulation result: generic binomially distributed outcome trial + * Undesirable outcome + * Initial/final common control arms: B/B + + Final status: conclusive, stopped for superiority + Final/maximum allowed sample sizes: 1500/2000 (75.0%) + Available outcome data at last adaptive analysis: 1500/1500 (100.0%) + + Trial results overview: + arms true_ys final_status status_look status_probs final_alloc + A 0.20 inferior 1500 0.0046 0.60 + B 0.15 superior 1500 0.9954 0.40 + C 0.30 inferior 1000 0.0010 0.15 + + Esimates from final analysis (all patients): + arms sum_ys_all ns_all raw_ests_all post_ests_all post_errs_all lo_cri_all + A 143 701 0.204 0.205 0.0152 0.176 + B 88 593 0.148 0.149 0.0151 0.122 + C 56 206 0.272 0.273 0.0310 0.213 + hi_cri_all + 0.235 + 0.179 + 0.336 + + Estimates from last adaptive analysis including each arm: + arms sum_ys ns raw_ests post_ests post_errs lo_cri hi_cri + A 143 701 0.204 0.205 0.0151 0.176 0.235 + B 88 593 0.148 0.149 0.0144 0.122 0.180 + C 56 206 0.272 0.274 0.0303 0.216 0.334 + + Simulation details: + * Random seed: 12345 + * Credible interval width: 95% + * Number of posterior draws: 5000 + * Posterior estimation method: medians with MAD-SDs + # dispatch_trial_runs works Code @@ -548,25 +589,25 @@ 18 18 2000 472 0.2360000 max 19 19 1600 372 0.2325000 superiority B B 20 20 500 135 0.2700000 superiority B B - sq_err sq_err_te - 1 1.403445e-03 NA - 2 6.546625e-05 NA - 3 8.482272e-05 NA - 4 6.368035e-04 NA - 5 1.355680e-05 NA - 6 1.071614e-03 NA - 7 1.475723e-06 NA - 8 NA NA - 9 1.059480e-05 NA - 10 NA NA - 11 1.336775e-04 NA - 12 1.254923e-03 NA - 13 NA NA - 14 NA NA - 15 3.127227e-04 NA - 16 NA NA - 17 NA NA - 18 NA NA - 19 2.154692e-05 NA - 20 4.407718e-05 NA + err sq_err err_te sq_err_te + 1 -0.037462579 1.403445e-03 NA NA + 2 -0.008091122 6.546625e-05 NA NA + 3 -0.009209925 8.482272e-05 NA NA + 4 -0.025234966 6.368035e-04 NA NA + 5 -0.003681956 1.355680e-05 NA NA + 6 -0.032735518 1.071614e-03 NA NA + 7 -0.001214794 1.475723e-06 NA NA + 8 NA NA NA NA + 9 -0.003254965 1.059480e-05 NA NA + 10 NA NA NA NA + 11 -0.011561899 1.336775e-04 NA NA + 12 -0.035424899 1.254923e-03 NA NA + 13 NA NA NA NA + 14 NA NA NA NA + 15 -0.017683969 3.127227e-04 NA NA + 16 NA NA NA NA + 17 NA NA NA NA + 18 NA NA NA NA + 19 0.004641866 2.154692e-05 NA NA + 20 0.006639065 4.407718e-05 NA NA diff --git a/tests/testthat/_snaps/summary-print.md b/tests/testthat/_snaps/summary-print.md index 9401a00..e6e8cf4 100644 --- a/tests/testthat/_snaps/summary-print.md +++ b/tests/testthat/_snaps/summary-print.md @@ -141,12 +141,12 @@ * Futility: 75.0% * Inconclusive at max sample size: 0.0% * Selection probabilities: A: 0.0% | B: 0.0% | C: 25.0% | None: 75.0% - * RMSE: 0.02621 - * RMSE treatment effect: 0.02309 + * RMSE / MAE: 0.02621 / 0.01865 + * RMSE / MAE treatment effect: 0.02309 / 0.02339 * Ideal design percentage: 100.0% Simulation details: - * Simulation time: 0.73 secs + * Simulation time: 1.05 secs * Base random seed: 12345 * Credible interval width: 95% * Number of posterior draws: 5000 @@ -175,12 +175,12 @@ * Futility: 75.0% * Inconclusive at max sample size: 0.0% * Selection probabilities: A: 0.0% | B: 0.0% | C: 25.0% | None: 75.0% - * RMSE: 0.02621 - * RMSE treatment effect: 0.02309 + * RMSE / MAE: 0.02621 / 0.01865 + * RMSE / MAE treatment effect: 0.02309 / 0.02339 * Ideal design percentage: 100.0% Simulation details: - * Simulation time: 0.73 secs + * Simulation time: 1.05 secs * Base random seed: 12345 * Credible interval width: 95% * Number of posterior draws: 5000 diff --git a/tests/testthat/setup_testdata_files.R b/tests/testthat/setup_testdata_files.R index 2f77fbb..f021749 100644 --- a/tests/testthat/setup_testdata_files.R +++ b/tests/testthat/setup_testdata_files.R @@ -58,6 +58,19 @@ if (FALSE) { res <- run_trials(trial, n_rep = 20, base_seed = 12345, sparse = FALSE) save_testdata(res, "binom__results__3_arms__common_control__equivalence__futility__softened") + # Binomial trial without common control, equivalence testing and actual stopping for equivalence + trial <- setup_trial_binom( + arms = c("A", "B"), + true_ys = c(0.25, 0.23), + fixed_probs = c(0.5, 0.5), + data_looks = 1:20 * 100, + randomised_at_looks = 1:20 * 100 + 100, + equivalence_prob = 0.8, + equivalence_diff = 0.03 + ) + res <- run_trials(trial, n_rep = 20, base_seed = 12345) + save_testdata(res, "binom__results__3_arms__no_control__equivalence__stopping") + # Normally distributed outcome trial with common control, "matched" control # group allocation, multiple best arms, varying probability thresholds, and # additional info (by default) @@ -97,5 +110,10 @@ if (FALSE) { res <- run_trials(trial, n_rep = 20, base_seed = 12345) save_testdata(res, "norm__results__3_arms__common_control__fixed__all_arms_fixed") + + # Calibration object (settings to facilitate speed) + trial <- setup_trial_binom(arms = c("A", "B"), true_ys = c(0.5, 0.5), data_looks = 100, n_draws = 1000) + res <- calibrate_trial(trial_spec = trial, search_range = c(0.95, 0.9999), tol = 0.05, base_seed = 12345) + save_testdata(res, "binom___calibration___setup2_arms__no_difference___rar") } diff --git a/tests/testthat/test-calibrate_trial.R b/tests/testthat/test-calibrate_trial.R index b996772..3e9807f 100644 --- a/tests/testthat/test-calibrate_trial.R +++ b/tests/testthat/test-calibrate_trial.R @@ -41,7 +41,7 @@ test_that("calibrate_trial works", { expect_identical(oldseed, get(".Random.seed", envir = globalenv())) }) -test_that("calibrate_trial errors/warns/messages correctly", { +test_that("calibrate_trial errors/warns correctly", { spec <- setup_trial_binom(arms = 1:2, true_ys = rep(0.35, 2), data_looks = 500 * 1:5) err_spec <- spec @@ -133,3 +133,10 @@ test_that("calibrate_trial errors/warns/messages correctly", { prev_x = 1, prev_y = 0, path = tmp_file))) }) + + +test_that("calibrate_trial messages correctly", { + spec <- suppressWarnings(setup_trial_binom(arms = 1:2, true_ys = rep(0.35, 2), data_looks = 100, n_draws = 100)) + expect_message(calibrate_trial(spec, target = 1, tol = 1)) + expect_message(calibrate_trial(spec, target = 1, tol = 1, base_seed = 123, n_rep = 100)) +}) diff --git a/tests/testthat/test-extract.R b/tests/testthat/test-extract.R index a7e4c51..5150ada 100644 --- a/tests/testthat/test-extract.R +++ b/tests/testthat/test-extract.R @@ -29,6 +29,11 @@ test_that("Tidy results of simulations from binomial-outcome trial without commo expect_snapshot(extract_results(res, raw_ests = TRUE)) }) +test_that("Selection works with practical equivalence and outcome-data lag works", { + res <- read_testdata("binom__results__3_arms__no_control__equivalence__stopping") # Sims with actual stopping for equivalence + expect_snapshot(extract_results(res, select_strategy = "best")) +}) + test_that("Sequential and parallel extraction works similarly", { res <- read_testdata("binom__results__3_arms__common_control__equivalence__futility__softened") expect_identical(extract_results(res, cores = 1), diff --git a/tests/testthat/test-gp_opt.R b/tests/testthat/test-gp_opt.R index 48d2a6a..3aedb91 100644 --- a/tests/testthat/test-gp_opt.R +++ b/tests/testthat/test-gp_opt.R @@ -29,4 +29,7 @@ test_that("gp_opt works", { lengthscale = c(0.1, 10))$next_x, 0.99378757515030058389) expect_equal(gp_opt(x, y, target = 0.1, resolution = 500, noisy = TRUE, lengthscale = 0.95)$next_x, 0.979759519038076120989) + expect_equal(gp_opt(x, y, target = 0.1, resolution = 500, noisy = TRUE, + lengthscale = 0.95, scale_x = FALSE)$next_x, + 0.97014028056112222576) }) diff --git a/tests/testthat/test-plot_metrics_ecdf.R b/tests/testthat/test-plot_metrics_ecdf.R index 35966c6..551e886 100644 --- a/tests/testthat/test-plot_metrics_ecdf.R +++ b/tests/testthat/test-plot_metrics_ecdf.R @@ -9,10 +9,21 @@ test_that("plot_metrics_ecdf works and errors correctly", { vdiffr::expect_doppelganger("selected", p) p <- plot_metrics_ecdf(res, metrics = "size") vdiffr::expect_doppelganger("size only", p) + p <- plot_metrics_ecdf(res, metrics = c("err", "sq_err", "err_te", "sq_err_te", + "abs_err", "abs_err_te"), te_comp = "A") + vdiffr::expect_doppelganger("errors", p) # Errors expect_error(plot_metrics_ecdf(res, metrics = TRUE)) expect_error(plot_metrics_ecdf(res, metrics = c("sum ys", "prob_conclusive"))) expect_error(plot_metrics_ecdf(res, metrics = c("sum ys", "sum_ys"))) expect_error(plot_metrics_ecdf(res, restrict = "positive trials")) + + spec_no_concl <- setup_trial_binom(arms = 1:2, true_ys = rep(0.1, 2), data_looks = 10) + res_no_concl <- run_trials(spec_no_concl, n_rep = 1, base_seed = 1) + expect_error(plot_metrics_ecdf(res_no_concl, restrict = "superior")) + + expect_error(plot_metrics_ecdf(res, metrics = "sq_err_te")) + expect_error(plot_metrics_ecdf(res, metrics = "err_te")) + expect_error(plot_metrics_ecdf(res, metrics = "abs_err_te")) }) diff --git a/tests/testthat/test-prob_funs.R b/tests/testthat/test-prob_funs.R index e58c300..089f565 100644 --- a/tests/testthat/test-prob_funs.R +++ b/tests/testthat/test-prob_funs.R @@ -97,4 +97,54 @@ test_that("reallocate_probs works", { reallocate_probs(probs_best, fixed_probs = c(0.4, 0.3, 0.3), min_probs = rep(NA, 3), max_probs = rep(NA, 3)), c(A = 0.4, B = 0.3, C = 0.3) ) + + # rescaling of of fixed probabilities/probability limits, various versions + + probs_best_rescale <- setNames(c(0.75, 0.1, 0.15), names(probs_best)) + expect_equal( + tolerance = 10^-6, + reallocate_probs(probs_best_rescale, fixed_probs = c(0.40, NA, NA), + min_probs = c(NA, 0.15, 0.15), max_probs = c(NA, 0.85, 0.85), + rescale_fixed = TRUE, rescale_limits = TRUE, rescale_factor = 4/3), + c(A = 0.5333333, B = 0.2, C = 0.2666667) + ) + + probs_best_rescale <- setNames(c(0.1, 0.15, 0.75), names(probs_best)) + expect_equal( + tolerance = 10^-6, + reallocate_probs(probs_best_rescale, fixed_probs = c(0.40, NA, NA), + min_probs = c(NA, 0.15, 0.15), max_probs = c(NA, 0.85, 0.85), + rescale_fixed = FALSE, rescale_limits = TRUE, + rescale_factor = 4/3), + c(A = 0.4, B = 0.2, C = 0.4) + ) + + probs_best_rescale <- setNames(c(0.1, 0.15, 0.75), names(probs_best)) + expect_equal( + tolerance = 10^-6, + reallocate_probs(probs_best_rescale, fixed_probs = c(0.40, NA, NA), + min_probs = c(NA, 0.15, 0.15), max_probs = c(NA, 0.85, 0.85), + rescale_fixed = TRUE, rescale_limits = FALSE, rescale_factor = 4/3), + c(A = 0.5333333, B = 0.15, C = 0.3166667) + ) + + # invalid combinations obtained by rescaling, but this is normalised + probs_best_rescale <- setNames(c(0.1, 0.9), c("A", "B")) + expect_equal( + tolerance = 10^-6, + reallocate_probs(probs_best_rescale, fixed_probs = c(0.40, NA), + min_probs = c(NA, 0.25), max_probs = c(NA, 0.75), + rescale_fixed = TRUE, rescale_limits = TRUE, rescale_factor = 4/2), + c(A = 0.6153846, B = 0.3846154) + ) + + probs_best_rescale <- setNames(c(0.1, 0.15, 0.75), names(probs_best)) + expect_equal( + tolerance = 10^-6, + reallocate_probs(probs_best_rescale, fixed_probs = c(0.40, NA, NA), + min_probs = c(NA, 0.15, 0.15), max_probs = c(NA, 0.85, 0.85), + rescale_fixed = TRUE, rescale_limits = TRUE, rescale_factor = 4/3, + rescale_ignore = 1), + c(A = 0.4, B = 0.2, C = 0.4) + ) }) diff --git a/tests/testthat/test-run_trials.R b/tests/testthat/test-run_trials.R index 613d0eb..60057a7 100644 --- a/tests/testthat/test-run_trials.R +++ b/tests/testthat/test-run_trials.R @@ -43,6 +43,20 @@ test_that("single trial simulation works", { ) expect_snapshot(run_trial(setup_equi_futil_only_first, seed = 12345)) + # Check that trial with rescaled probabilities works + setup_rescale_probs <- setup_trial_binom( + arms = c("A", "B", "C"), + control = "B", + start_probs = c(0.4, 0.3, 0.3), + fixed_probs = c(0.4, NA, NA), + min_probs = c(NA, 0.15, 0.15), + max_probs = c(NA, 0.85, 0.85), + rescale_probs = "both", + true_ys = c(0.2, 0.15, 0.3), + data_looks = seq(from = 500, to = 2000, by = 500) + ) + expect_snapshot(run_trial(setup_rescale_probs, seed = 12345)) + # Check that seed is unchanged expect_identical(oldseed, get(".Random.seed", envir = globalenv())) }) diff --git a/tests/testthat/test-setup_trial.R b/tests/testthat/test-setup_trial.R index ab118b3..3dea437 100644 --- a/tests/testthat/test-setup_trial.R +++ b/tests/testthat/test-setup_trial.R @@ -157,18 +157,20 @@ test_that("validate setup trial specifications", { test_that("setup/validate_trial functions errors on invalid inputs", { expect_error(validate_trial(arms = NULL)) expect_error(validate_trial(arms = c("A", "A", "B"))) + expect_error(validate_trial(arms = "A")) expect_error(validate_trial(arms = c(1, 2, 3), control = 1)) - expect_error(validate_trial(arms = c("A", "B", "C"), control_prob_fixed = 0.4)) + expect_error(validate_trial(arms = c("A", "B", "C"), control_prob_fixed = 0.4, + data_looks = 1:5 * 100)) expect_error(validate_trial(arms = c("A", "B", "C"), control = "A", control_prob_fixed = "sqrt-based", start_probs = rep(1/3, 3))) expect_error(validate_trial(arms = c("A", "B", "C"), control = "A", control_prob_fixed = "sqrt-based fixed", fixed_probs = rep(1/3, 3))) expect_error(validate_trial(arms = c("A", "B", "C"), control = "A", control_prob_fixed = "sqrt-based start", fixed_probs = rep(1/3, 3))) - expect_error(validate_trial(arms = c("A", "B", "C"), control = "A", control_prob_fixed = "match", - fixed_probs = c(0.3, 0.3, 0.4))) expect_error(validate_trial(arms = c("A", "B", "C"), control = "A", control_prob_fixed = "match", start_probs = c(0.3, 0.3, 0.4))) + expect_error(validate_trial(arms = c("A", "B", "C"), control = "A", control_prob_fixed = "match", + fixed_probs = c(1/3, NA, NA), data_looks = 1:5 * 100)) expect_error(validate_trial(arms = c("A", "B", "C"), start_probs = rep(0.25, 4))) expect_error(validate_trial(arms = 1:3, start_probs = rep(0.32, 3))) @@ -181,6 +183,16 @@ test_that("setup/validate_trial functions errors on invalid inputs", { expect_error(validate_trial(arms = 1:3, start_probs = c(0.5, 0.25, 0.25), min_probs = c(0.5, 0.1, 0.1), max_probs = c(0.5, NA, NA))) + expect_error(validate_trial(arms = 1:3, rescale_probs = "invalid")) + expect_error(validate_trial(arms = 1:3, rescale_probs = c("fixed", "both"))) + expect_error(validate_trial(arms = 1:2, rescale_probs = "both")) + expect_error(validate_trial(arms = 1:3, control = 1, control_prob_fixed = "sqrt-based fixed", + rescale_probs = "fixed", data_looks = 1:5 * 100)) + expect_error(validate_trial(arms = 1:3, rescale_probs = "fixed")) + expect_error(validate_trial(arms = 1:3, control = 1, control_prob_fixed = "sqrt-based", + rescale_probs = "fixed")) + expect_error(validate_trial(arms = 1:3, rescale_probs = "limits")) + expect_error(validate_trial(arms = 1:3, data_looks = c(100, 100, 200))) expect_error(validate_trial(arms = 1:3, data_looks = c(100, 200, 300), look_after_every = 100, max_n = 300)) expect_error(validate_trial(arms = 1:3)) @@ -255,6 +267,7 @@ test_that("setup/validate_trial functions errors on invalid inputs", { futility_prob = 0.9, futility_diff = 2, futility_only_first = TRUE)) expect_error(setup_trial_norm(arms = 1:3, data_looks = 1:3 * 100, true_ys = 1:3, sds = -1)) + expect_error(setup_trial_norm(arms = 1:3, data_looks = 1:3 * 100)) expect_error(setup_trial_binom(arms = 1:3, max_n = 28.9, look_after_every = 1.23, true_ys = 1:3 * 0.1)) expect_error(setup_trial_binom(arms = 1:3, true_ys = 1:3 * 0.1, data_looks = 100 / 3 * 1:3)) diff --git a/tests/testthat/test-update_saved_trials.R b/tests/testthat/test-update_saved.R similarity index 51% rename from tests/testthat/test-update_saved_trials.R rename to tests/testthat/test-update_saved.R index e1cbba7..c5eae3a 100644 --- a/tests/testthat/test-update_saved_trials.R +++ b/tests/testthat/test-update_saved.R @@ -1,4 +1,4 @@ -test_that("updating outdated trials objects works", { +test_that("updating outdated trial_results objects works", { tmp_file <- tempfile() on.exit(try(file.remove(tmp_file)), add = TRUE, after = FALSE) @@ -10,6 +10,7 @@ test_that("updating outdated trials objects works", { # Test v1.1.1 or before pseudo_old_res <- res pseudo_old_res$adaptr_version <- NULL # mimic what happened in adaptr until v1.1.1 + pseudo_old_res$trial_spec$rescale_probs <- NULL saveRDS(res, tmp_file) expect_warning(update_saved_trials(tmp_file)) @@ -26,10 +27,38 @@ test_that("updating outdated trials objects works", { saveRDS(1:10, tmp_file) expect_error(update_saved_trials(tmp_file)) - # Test v1.2.0 + # Test v1.2.0+ pseudo_old_res <- res + pseudo_old_res$trial_spec$rescale_probs <- NULL pseudo_old_res$adaptr_version <- as.package_version("1.2.0") saveRDS(pseudo_old_res, tmp_file) expect_identical(update_saved_trials(tmp_file), res) }) + +test_that("updating outdated trial_calibration objects works", { + tmp_file <- tempfile() + on.exit(try(file.remove(tmp_file)), add = TRUE, after = FALSE) + + expect_error(update_saved_calibration("invalid_fpath.adaptr")) + expect_error(update_saved_calibration(tmp_file)) + + res <- read_testdata("binom___calibration___setup2_arms__no_difference___rar") + + # Test v1.3.0+ + pseudo_old_res <- res + pseudo_old_res$adaptr_version <- as.package_version("1.3.0") + pseudo_old_res$input_trial_spec$rescale_probs <- NULL + pseudo_old_res$best_trial_spec$rescale_probs <- NULL + pseudo_old_res$best_sims$adaptr_version <- as.package_version("1.3.0") + + saveRDS(res, tmp_file) + expect_warning(update_saved_calibration(tmp_file)) + + saveRDS(pseudo_old_res, tmp_file) + expect_invisible(update_saved_calibration(tmp_file)) + expect_warning(update_saved_calibration(tmp_file)) + + saveRDS(1:10, tmp_file) + expect_error(update_saved_calibration(tmp_file)) +}) diff --git a/vignettes/Basic-examples.Rmd b/vignettes/Basic-examples.Rmd index b2fe903..91fffa0 100644 --- a/vignettes/Basic-examples.Rmd +++ b/vignettes/Basic-examples.Rmd @@ -214,7 +214,9 @@ setup_trial_binom( # Minimum probabilities of 20% for non-control arms, must be NA for the # control arm with fixed allocation probability # Limits are ignored for arms that become subsequent controls + # Limits are rescaled (i.e., increased proportionally) when arms are dropped min_probs = c(NA, 0.2, 0.2, 0.2), + rescale_probs = "limits", # Constant softening of 0.5 (= square-root transformation) soften_power = 0.5 @@ -346,3 +348,24 @@ setup_trial_binom( ) ``` +### Example 9: minimum allocation probabilities rescaled when arms are dropped + +In this example, a trial design with four arms and restricted RAR (minimum +allocation limits) is specified, with additional specification that the minimum +allocation limits should be rescaled proportionally when arms are dropped +(rescaling can similarly be applied to fixed allocation probabilities): + +```{r} +setup_trial_binom( + arms = c("A", "B", "C", "D"), + control = "A", + true_ys = c(0.2, 0.2, 0.2, 0.2), + + min_probs = rep(0.15, 4), # Specify initial minimum allocation probabilities + # Rescale allocation probability limits as arms are dropped + rescale_probs = "limits", + + data_looks = seq(from = 100, to = 1000, by = 100) +) +``` + diff --git a/vignettes/Overview.Rmd b/vignettes/Overview.Rmd index 435e127..47130a5 100644 --- a/vignettes/Overview.Rmd +++ b/vignettes/Overview.Rmd @@ -21,69 +21,81 @@ using adaptive stopping, adaptive arm dropping and/or response-adaptive randomisation. The package has been developed as part of the -[INCEPT (Intensive Care Platform Trial) project](https://incept.dk/), funded -primarily by a grant from -[Sygeforsikringen "danmark"](https://www.sygeforsikring.dk/). +[INCEPT (Intensive Care Platform Trial) project](https://incept.dk/), primarily +supported by a grant from [Sygeforsikringen "danmark"](https://www.sygeforsikring.dk/). Additional guidance on the key methodological considerations when planning and comparing adaptive clinical trials can be found in the open access article *"[An overview of methodological considerations regarding adaptive stopping, arm dropping and randomisation in clinical trials](https://doi.org/10.1016/j.jclinepi.2022.11.002)"* available in Journal of Clinical Epidemiology. -## Basic example -First, load the package: +## Usage and workflow overview + +The central functionality of `adaptr` and the typical workflow is illustrated +here. + +### Setup + +First, the package is loaded and a cluster of parallel workers is initiated by +the `setup_cluster()` function to facilitate parallel computing: ```{r} library(adaptr) + +setup_cluster(2) ``` Parallelisation is supported in many `adaptr` functions, and a cluster of -parallel workers may be setup for the entire session using `setup_cluster()`, -which is ideally called early in a script. Alternatively, parallelisation +parallel workers can be setup for the entire session using `setup_cluster()` +early in the script as in this example. Alternatively, parallelisation can be controlled by the global `"mc.cores"` option (set by calling `options(mc.cores = )`) or the `cores` argument of many functions. -### Set up trial -Then, setup a trial with the desired specifications. `adaptr` offers the -general purpose function `setup_trial()`, but here we use the built-in -`setup_trial_binom()` for a trial with a binary, binomially distributed, -undesirable outcome such as mortality (`adaptr` also includes -`setup_trial_norm()` for continuous, normally distributed outcomes). +### Specify trial design -The example trial specification has the following characteristics: - -- The allocation probability to each arm cannot be lower than 15% (`min_probs`). -- Default thresholds for `inferiority` (< 1% probability of being the best arm) -and `superiority` (> 99% probability of being the best arm) are used and hence -not specified. -- Equivalence stopping rule: if the simulation yields a 90% probability -(`equivalence_prob`) of treatment differences being < 5 %-points -(`equivalence_diff`), the trial is stopped. -- We soften allocation ratios (`soften_power`) by a constant factor. +Setup a trial specification (defining the trial design and scenario) using +the general `setup_trial()` function, or one of the special case variants using +default priors `setup_trial_binom()` (for binary, binomially distributed +outcomes; used in this example) or `setup_trial_norm()` (for continuous, +normally distributed outcomes). -```{r} -binom_trial <- setup_trial_binom( - arms = c("Arm A", "Arm B", "Arm C"), - true_ys = c(0.25, 0.20, 0.30), - min_probs = rep(0.15, 3), - data_looks = seq(from = 300, to = 2000, by = 100), - equivalence_prob = 0.9, - equivalence_diff = 0.05, - soften_power = 0.5 -) -``` +The example trial specification has the following characteristics: -See `?setup_trial()` for more details on the arguments or +- A binary, binomially distributed, undesirable (default) outcome +- Three arms with no designated common control +- Identical underlying outcome probabilities of 25% in each arm +- Analyses conducted when specific number of patients have outcome data +available, with more patients randomised at all but the last look (lag due to +follow-up and data collection/verification) +- No explicitly defined stopping thresholds for `inferiority` or `superiority` +(default thresholds of < 1% and > 99%, respectively, apply) +- Equivalence stopping rule defined as > 90% probability (`equivalence_prob`) of +between-arm differences of all remaining arms being < 5 %-points +- Response-adaptive randomisation with minimum allocation probabilities of 20% +and softening of allocation ratios by a constant factor (`soften_power`) + +See `?setup_trial()` for details on all the arguments or `vignette("Basic-examples", "adaptr")` for **basic** example trial specifications and a thorough review of the general trial specification settings, and `vignette("Advanced-example", "adaptr")` for an **advanced** example including details on how to specify user-written functions for generating outcomes and posterior draws. -We can print an overview of the trial specification by simply running: +Below, the trial specification is setup and a human-readable overview printed: ```{r} -binom_trial +binom_trial <- setup_trial_binom( + arms = c("Arm A", "Arm B", "Arm C"), + true_ys = c(0.25, 0.25, 0.25), + min_probs = rep(0.20, 3), + data_looks = seq(from = 300, to = 2000, by = 100), + randomised_at_looks = c(seq(from = 400, to = 2000, by = 100), 2000), + equivalence_prob = 0.9, + equivalence_diff = 0.05, + soften_power = 0.5 +) + +print(binom_trial, prob_digits = 3) ``` By default, (most) probabilities are shown with 3 decimals. This can be changed @@ -94,165 +106,290 @@ for example: print(binom_trial, prob_digits = 2) ``` -Finally, a trial specification may be calibrated to obtain a specific value for -a certain performance metric (e.g., the Bayesian type 1 error rate for trial -specifications with no between-arm differences; not done in this overview) by -using the `calibrate_trial()` function. - -### Simulate a single trial -Remember to define the `seed` to ensure reproducible results: +### Calibration + +In the example trial specification, there are no true between-arm differences, +and stopping rules for inferiority and superiority are not explicitly defined. +This is intentional, as these stopping rules will be calibrated to obtain a +desired probability of stopping for superiority in the scenario with no +between-arm differences (corresponding to the Bayesian type 1 error rate). Trial +specifications do not necessarily have to be calibrated. Instead,simulations can +be run directly using the `run_trials()` function covered below (or +`run_trial()` for a single simulation). This can be followed by assessment of +performance metrics, and manually changing the specification (including the +stopping rules) until performance metrics are considered acceptable. In this +example, a full calibration procedure is performed. + +Calibration of a trial specification is done using the `calibrate_trial()` +function, which defaults to calibrate constant, symmetrical stopping rules +for inferiority and superiority (expecting a trial specification with +identical outcomes in each arm), but can be used to calibrate any parameter in a +trial specification towards any performance metric if a user-defined calibration +function (`fun`) is specified. + +To perform the calibration, a `target` value, a `search_range`, a tolerance +value (`tol`), and the allowed direction of the tolerance value (`dir`) must be +specified (or alternatively, the defaults can be used). Of note, the number of +simulations in each calibration step here is lower than generally recommended +(to reduce the time required to build this vignette): ```{r} -trial_res <- run_trial(binom_trial, seed = 12345) +# Calibrate the trial specification +calibrated_binom_trial <- calibrate_trial( + trial_spec = binom_trial, + n_rep = 1000, # 1000 simulations for each step (more generally recommended) + base_seed = 4131, # Base random seed (for reproducible results) + target = 0.05, # Target value for calibrated metric (default value) + search_range = c(0.9, 1), # Search range for superiority stopping threshold + tol = 0.01, # Tolerance range + dir = -1 # Tolerance range only applies below target +) -trial_res +# Print result (to check if calibration is successful) +calibrated_binom_trial ``` -Again, we can choose the number of decimals with `print()`: +The calibration is successful (if not, results should not be used, and the +calibration settings should be changed and the calibration repeated). +The calibrated, constant stopping threshold for superiority is printed with the +results (`r calibrated_binom_trial$best_x`) and can be extracted using +`calibrated_binom_trial$best_x`. Using the default calibration functionality, +the calibrated, constant stopping threshold for inferiority is symmetrical, +i.e., `1 - stopping threshold for superiority` +(`r 1 - calibrated_binom_trial$best_x`). The calibrated trial specification +may be extracted using `calibrated_binom_trial$best_trial_spec` and, if printed, +will also include the calibrated stopping thresholds. + +Calibration results may be saved and reloaded by using the `path` argument, to +avoid unnecessary repeated simulations. + + +### Summarising results + +The results of the simulations using the calibrated trial specification +conducted during the calibration procedure may be extracted using +`calibrated_binom_trial$best_sims`. These results can be summarised with several +functions. Most of these functions support different 'selection strategies' for +simulations not ending with superiority, i.e., performance metrics can be +calculated assuming different arms would be used in clinical practice if no arm +is ultimately superior. + +The `check_performance()` function summarises performance metrics in a tidy +`data.frame`, with uncertainty measures (bootstrapped confidence intervals) if +requested. Here, performance metrics are calculated considering the 'best' arm +(i.e., the one with the highest probability of being overall best) selected in +simulations not ending with superiority: ```{r} -print(trial_res, prob_digits = 2) +# Calculate performance metrics with uncertainty measures +binom_trial_performance <- check_performance( + calibrated_binom_trial$best_sims, + select_strategy = "best", + uncertainty = TRUE, # Calculate uncertainty measures + n_boot = 1000, # 1000 bootstrap samples (more typically recommended) + ci_width = 0.95, # 95% confidence intervals (default) + boot_seed = "base" # Use same random seed for bootstrapping as for simulations +) + +# Print results +print(binom_trial_performance, digits = 2) ``` -### Simulate multiple trials -Generally, we want to run many simulations using the same trial specification to -assess and compare performance metrics of different trial designs. This is the -job of `run_trials()` (note the final ***s***); again, we specify a `base_seed` -for reproducible results. Here we run 25 simulations, but in practice you will -generally want to run more simulations. The low number of simulations in this -example has been chosen to make run-time tolerable when producing the example, -but leads to uncertainty and instability in the results, as seen below. +Similar results in `list` format (without uncertainty measures) can be obtained +using the `summary()` method (as known from, e.g., regression models in`R`), +which comes with a `print()` method providing formatted results. If the +simulation results are printed directly, this function is called with the +default arguments (all arguments, e.g., selection strategies may also be +directly supplied to the `print()` method). ```{r} -trial_res_mult <- run_trials(binom_trial, n_rep = 25, base_seed = 67890) -``` +binom_trial_summary <- summary( + calibrated_binom_trial$best_sims, + select_strategy = "best" +) -`run_trials()` can run simulations on several CPU cores concurrently by using -the `setup_cluster()` function (recommended), the `"mc.cores"` global option, or -the `cores` argument of `run_trials()`, as mentioned above (further details in -the `setup_cluster()` and `run_trials()` function definitions). Most functions -used to post-process, extract, and plot results can similarly be run in -parallel. +print(binom_trial_summary, digits = 2) +``` -### Calculate performance metrics and summmarise results -The results of multiple simulations may be summarised by printing the resulting -object: +Individual simulation results can be extracted in a tidy `data.frame` using +`extract_results()`: ```{r} -trial_res_mult -``` +binom_trial_results <- extract_results( + calibrated_binom_trial$best_sims, + select_strategy = "best" +) -This calls the `summary()` method (as known from, e.g., regression models in -`R`), which summarises the results, and prints the output from that function in -a human-friendly manner using the `print()` method for summarised simulations. +nrow(binom_trial_results) # Number of rows/simulations -The `summary()` function can also be called directly, which allows more control -of how results are summarised (including which arms are selected in inconclusive -trials), and allows subsequent extraction of individual key results. In -addition, the number of digits can be controlled when printed: +head(binom_trial_results) # Print the first rows +``` -```{r} -res_sum <- summary(trial_res_mult) +Finally, the probabilities of different remaining arms and +their statuses (with uncertainty) at the last adaptive analysis can be +summarised using the `check_remaining_arms()` function (dropped arms will +be shown with an empty text string): -print(res_sum, digits = 1) +```{r} +check_remaining_arms( + calibrated_binom_trial$best_sims, + ci_width = 0.95 # 95% confidence intervals (default) +) ``` -The `summary()` method, however, may not necessarily be what you want. `adaptr` -has additional functions that may be used on multiple simulation results of the -same trial specification. +### Visualising results -The `extract_results()` function extract key trial results and yields a tidy -`data.frame` with one simulation per row: +Several visualisation functions are included (all are optional, and all require +the `ggplot2` package installed). + +Convergence and stability of one or more performance metrics may be visually +assessed using `plot_convergence()` function: ```{r} -extr_res <- extract_results(trial_res_mult) +plot_convergence( + calibrated_binom_trial$best_sims, + metrics = c("size mean", "prob_superior", "prob_equivalence"), + # select_strategy can be specified, but does not affect the chosen metrics +) +``` -nrow(extr_res) +Plotting other metrics is possible; see the `plot_convergence()` documentation. +The simulation results may also be split into separate, consecutive batches when +assessing convergence, to further assess the stability: -head(extr_res) +```{r} +plot_convergence( + calibrated_binom_trial$best_sims, + metrics = c("size mean", "prob_superior", "prob_equivalence"), + n_split = 4 +) ``` -The `check_performance()` function calculates key performance metrics and -returns them in a tidy `data.frame` with one metric per row, and may also be -used to assess the uncertainty in these estimates using bootstrapping: +The status probabilities for the overall trial according to trial progress can +be visualised using the `plot_status()` function: ```{r} -perf_res <- check_performance(trial_res_mult, uncertainty = TRUE, n_boot = 1000, - boot_seed = "base") - -print(perf_res, digits = 3) +plot_status( + calibrated_binom_trial$best_sims, + x_value = "total n" # Total number of randomised patients at X-axis +) ``` -The `plot_metrics_ecdf()` function plots empirical cumulative distribution -functions of numerical performance metrics: +Similarly, the status probabilities for one or more specific trial arms can be +visualised: ```{r} -plot_metrics_ecdf(trial_res_mult) +plot_status( + calibrated_binom_trial$best_sims, + x_value = "total n", + arm = NA # NA for all arms or character vector for specific arms +) ``` -Note that all `adaptr` plotting functions require the `ggplot2` package. +Finally, various metrics may be summarised over the progress of one or multiple +trial simulations using the `plot_history()` function, which requires non-sparse +results (the `sparse` argument must be `FALSE` in `calibrate_trials()`, +`run_trials()`, or `run_trial()`, leading to additional results being saved - +all other functions work with sparse results). This will be illustrated below. -The stability of performance metrics according to the number of simulations may -be assessed visually using the `plot_convergence()` function. +### Use calibrated stopping thresholds in another scenario + +The calibrated stopping thresholds (calibrated in a scenario with no between-arm +differences) may be used to run simulations with the same overall trial +specification, but according to a different scenario (i.e., with between-arm +differences present) to assess performance metrics (including the Bayesian +analogue of power). + +First, a new trial specification is setup using the same settings as before, +except for between-arm differences and the calibrated stopping thresholds: ```{r} -# Convergence plots for four performance metrics -plot_convergence(trial_res_mult, metrics = c("size mean", "prob superior", - "rmse", "idp")) +binom_trial_calib_diff <- setup_trial_binom( + arms = c("Arm A", "Arm B", "Arm C"), + true_ys = c(0.25, 0.20, 0.30), # Different outcomes in the arms + min_probs = rep(0.20, 3), + data_looks = seq(from = 300, to = 2000, by = 100), + randomised_at_looks = c(seq(from = 400, to = 2000, by = 100), 2000), + # Stopping rules for inferiority/superiority explicitly defined + # using the calibration results + inferiority = 1 - calibrated_binom_trial$best_x, + superiority = calibrated_binom_trial$best_x, + equivalence_prob = 0.9, + equivalence_diff = 0.05, + soften_power = 0.5 +) ``` -It is seen that the low number of simulations used in this example leads to -substantial uncertainty and instability in performance metrics (while the ideal -design percentage is always at 100% here, it would likely drop somewhat if more -simulations were conducted). - -Finally, all combinations of remaining arms after trial completion may be -summarised using `check_remaining_arms()`: +Simulations using the trial specification with calibrated stopping thresholds +and differences present can then be conducted using the `run_trials()` function. +Here, we specify that non-sparse results will be returned (to illustrate the +`plot_history()` function). ```{r} -check_remaining_arms(trial_res_mult) +binom_trial_diff_sims <- run_trials( + binom_trial_calib_diff, + n_rep = 1000, # 1000 simulations (more generally recommended) + base_seed = 1234, # Reproducible results + sparse = FALSE # Return additional results for visualisation +) ``` +Again, simulations may be saved and reloaded using the `path` argument. + +We then calculate performance metrics as above: -### Visualise trial results -In addition to the convergence plots, results may be visualised using the -`plot_status()` and `plot_history()` functions. -We need non-sparse results for `plot_history()` (but *not* for `plot_status()` -or `plot_convergence()` presented above), so we re-run `run_trials()` with the -`sparse` argument set to `FALSE`: +```{r} +check_performance( + binom_trial_diff_sims, + select_strategy = "best", + uncertainty = TRUE, + n_boot = 1000, # 1000 bootstrap samples (more typically recommended) + ci_width = 0.95, + boot_seed = "base" +) +``` + +Similarly, overall trial statuses for the scenario with differences are +visualised: ```{r} -trial_res_mult <- run_trials(binom_trial, n_rep = 25, base_seed = 67890, - sparse = FALSE) +plot_status(binom_trial_diff_sims, x_value = "total n") ``` -First, we plot the overall trial statuses according to the total number of -patients randomised (this function does *not* require `sparse = FALSE`): +Statuses for each arm in this scenario are also visualised: ```{r} -plot_status(trial_res_mult, x_value = "total n") +plot_status(binom_trial_diff_sims, x_value = "total n", arm = NA) ``` -We can also plot the statuses for specific arms, or for all arms if supplying -`NA` to the `arm`-argument: +We can plot the median and interquartile ranges of allocation probabilities in +each arm over time using the `plot_history()` function (requiring non-sparse +results, leading to substantially larger objects and files if saved): ```{r} -plot_status(trial_res_mult, x_value = "total n", arm = NA, ncol = 1) +plot_history( + binom_trial_diff_sims, + x_value = "total n", + y_value = "prob" +) ``` -Next, we plot the history of allocation probabilities at each adaptive -analysis look. Intervals cover the inter-quartile range by default -(`interval_width = 0.5`): +Similarly, the median (interquartile range) number of patients allocated to each +arm as the trial progresses can be visualised: ```{r} -plot_history(trial_res_mult) +plot_history( + binom_trial_diff_sims, + x_value = "total n", + y_value = "n all" +) ``` -Plotting other summary metrics is possible; see the `plot_history()` -documentation. +Plotting other metrics is possible; see the `plot_history()` documentation. ## Citation -If using the package, please consider citing: + +If you use the package, please consider citing it: ```{r} citation(package = "adaptr")