CI benchmark: add docs (#1250)

* theme, more N

* produce table instead of plot

* install tinytable

* produce table with more benchmarks, use 75k obs and 20 iterations

* add docs

* typos

.github/benchmark_docs.md

@@ -0,0 +1,70 @@
+Every time a PR is opened and a commit is added to the PR, a Github Actions 
+workflow is run with a suite of benchmarks. This document explains how this
+workflow works.
+
+# Defining the workflow
+
+The workflow running in Github Actions is stored in 
+`.github/workflows/benchmarks.yaml`. This runs on every pull request, every time
+there is a new commit that modifies a file in `R`, `src`, or `.github/scripts`.
+This is to avoid running benchmarks for changes in the README for instance.
+After installing required dependencies to run the benchmark (see next section),
+the workflow runs the benchmarks, stores a table containing the results, and
+automatically adds a comment in the PR with those results. To avoid flooding the
+PR with new comments every time a commit is pushed, only one comment is added
+and is then updated by new commits.
+
+One thing that makes this workflow slightly more challenging is that we need
+the workflow to have the permission to write (meaning using the Github token
+associated with it) on our own PRs as well as external PRs introduced via forks.
+To avoid giving forks access to our Github token, we will use 
+`on: pull_request_target` instead of `on: pull_request`. This will still trigger
+the workflow when a new commit is pushed in the PR, but the workflow will run
+against the `main` branch and not the against the PR, meaning that any changes
+in the workflow made in the PR will not run in Github actions while the PR is
+not merged.
+
+This is nicely explained in this blog post: https://jacobtomlinson.dev/posts/2022/commenting-on-pull-requests-with-github-actions/
+
+However this also means that if you open a PR to update 
+`.github/workflows/benchmarks.yaml`, then you will need to merge this PR to
+see the changes applied in the future workflows. You will not be able to see
+the changes "live" in the PR you opened.
+
+
+# Defining the benchmarks
+
+The R code with the benchmarks is stored in `.github/scripts/benchmarks.R`. It
+uses the R packages [`cross`](https://github.com/davisVaughan/cross) and 
+[`bench`](https://cran.r-project.org/web/packages/bench/):
+
+* `bench` computes the time and memory used for each expression;
+* `cross` allows one to run the same code on several packages, including those
+  corresponding to specific commits or branches. The following lines specify
+  that we want to run the benchmarks on the CRAN version, the `main` branch of
+  the development version, and the current PR in which this workflow runs:
+  ```r
+  pkgs = c(
+    "marginaleffects",
+    "vincentarelbundock/marginaleffects",
+    paste0("vincentarelbundock/marginaleffects#", pr_number)
+  )
+  ```
+
+Benchmarks are defined and run in `cross::run()`, which will take care of running
+them on all the versions mentioned above. It creates a nested data.frame
+with the timings, memory used, and more information for all expressions and
+all versions.
+
+The rest of the code cleans this data and computes comparisons between the PR 
+and main and between the PR and the CRAN version. Finally, it creates the 
+Markdown table and the rest of the content that will go in the comment that is 
+automatically posted in the pull request.
+
+# Adding benchmarks
+
+Adding benchmarks can be done in the `bench::mark()` call in 
+`.github/scripts/benchmarks.R`, but the more benchmarks there are, the longest
+the workflow will take. Generally speaking, the time spent by the workflow 
+depends on the number of benchmarks, the number of observations, and the 
+number of iterations, so there is a tradeoff.

.github/scripts/benchmarks.R

@@ -12,7 +12,6 @@ out <- cross::run(
   ),
   ~ {
     library(marginaleffects)
-    library(data.table)
 
     bench::press(
       N = 75000,
@@ -88,7 +87,8 @@ final <- unnested |>
   reshape(
     direction = "wide",
     idvar = "expression",
-    timevar = "pkg") |>
+    timevar = "pkg"
+  ) |>
   transform(
     # Compute change in time and memory between PR/main branch and PR/CRAN
     median_diff.main.pr = round((median.PR - median.main) / median.main * 100, 2),
@@ -101,22 +101,22 @@ final <- unnested |>
 
 cols <- c("median_diff.main.pr", "median_diff.CRAN.pr", "mem_alloc_diff.main.pr", "mem_alloc_diff.CRAN.pr")
 for (col in cols) {
-    old <- final[[col]]
-    new <- rep(NA_character_, nrow(final))
-    new <- ifelse(old >= 5, paste0(":collision: ", old, "%"), new)
-    new <- ifelse(old < 5 & old > -5, paste0(old, "%"), new)
-    new <- ifelse(old <= -5, paste0(":zap: ", old, "%"), new)
-    final[[col]] <- new
+  old <- final[[col]]
+  new <- rep(NA_character_, nrow(final))
+  new <- ifelse(old >= 5, paste0(":collision: ", old, "%"), new)
+  new <- ifelse(old < 5 & old > -5, paste0(old, "%"), new)
+  new <- ifelse(old <= -5, paste0(":zap: ", old, "%"), new)
+  final[[col]] <- new
 }
 
 cols <- c(
-    'Expression' = 'expression',
-    'PR time (median, seconds)' = 'median.PR',
-    '% change with "main"' = 'median_diff.main.pr',
-    '% change with CRAN' = 'median_diff.CRAN.pr',
-    'PR memory (MB)' = 'mem_alloc.PR',
-    'Mem. % change with "main"' = 'mem_alloc_diff.main.pr',
-    'Mem. % change with CRAN' = 'mem_alloc_diff.CRAN.pr'
+  "Expression" = "expression",
+  "PR time (median, seconds)" = "median.PR",
+  '% change with "main"' = "median_diff.main.pr",
+  "% change with CRAN" = "median_diff.CRAN.pr",
+  "PR memory (MB)" = "mem_alloc.PR",
+  'Mem. % change with "main"' = "mem_alloc_diff.main.pr",
+  "Mem. % change with CRAN" = "mem_alloc_diff.CRAN.pr"
 )
 
 final <- setNames(final[, cols], names(cols))

.github/workflows/benchmarks.yaml

@@ -44,7 +44,7 @@ jobs:
 
       - name: Install required dependencies for benchmark
         run: |
-          install.packages(c("remotes", "bench", "dplyr", "Formula", "pandoc", "tidyr", "tinytable"))
+          install.packages(c("remotes", "bench", "Formula", "pandoc", "tinytable"))
           remotes::install_github("DavisVaughan/cross")
         shell: Rscript {0}