Benchmark repository

[andrewiggins]

Problem

There are 3 challenges I see with how we do benchmarking today.

1. Scattered benchmark infra

Currently, we have benchmarks scattered across our various repositories (preact, signals, render-to-string, others). Some of these use copies the same infra (e.g. tachometer) and scripts while others are hand written. Keeping these up to date and in-sync is becoming more difficult and time-consuming.

2. Not all Preact "styles" are covered

There are different "styles" Preact apps can be written (e.g. class components, hooks, or using compat). Today, the preact repo only has implementations that test the core library which only natively supports class components. Changes to hooks or compat won't impact our current benchmarks and so we can't get feedback on how changes to those libraries impact our performance metrics.

3. Can't compare across implementations

Building on the previous challenge, we don't have infrastructure to compare the same app built using different "styles" (e.g. hooks vs class vs compat). This challenge is particularly felt for signals, which lives in a different repository from Preact core. This lack of infra prevents us from having data to make decisions about what to keep in core or what to move our core. Perhaps bringing a feature into core would improve performance for that use case in a significant way. Or moving a feature out of core and into compat would improve core's performance in a meaningful way. Being able to get more data on those kinds of changes would be helpful.

In moving our benchmarks to their own repo, I'd like to address each of these.

Key updates

To address these problems, the 2 big changes we are making to our benchmark infrastructure are below:

1. Move benchmarks into a separate repository

This new repository will be the central place for most of our benchmarks. Individual implementation repositories (e.g. preact, signals, preact-render-to-string) can checkout this repository in their CI builds to run the benchmarks against local changes in PRs. The scripts in this repository will also provide a facility to run benchmarks against local enlistments on your machine to make local development easy.

2. Add implementation pivot to scripts

Our benchmark scripts will be updated to understand that different implementations may exist for each benchmark. So for one benchmark (e.g. 02_replace1k.html), an implementation may exist using preact classes, hooks, compat and signals. When running the benchmarks, the user can choose which versions of backing libraries (e.g. preact-master, preact-signals-local) and which implementations (signals and hooks) to run.

[andrewiggins]

Folder structure

Proposed folder structure:

- versions/
  - preact/
    - local/
    - master/
  - signals/
    - local/
    - master/
  - ...
- scripts/
- src/
  - table-app/
    - _shared/
    - preact-class/
    - preact-hooks/
    - preact-compat/
    - ...
    - 02_replace1k.html
    - 03_update1k10k.html
  - movie-app/ ...
  - todo/ ...
  - filter-list/ ...

Here, the src folder contains the implementations of various benchmarks. We'll organize the src folder by "app" first, then implementation so different implementations of the same app can share details (simple functional components or button IDs) between each other. Within each app folder, the individual benchmarks will be in the .html files. The code in the html files will load the app, prep, and run the benchmark.

The versions folder will contain packages for different dependencies of the benchmark implementations. Some (maybe most) will be direct proxy exports of the package. But others will expose API adapters to massage over minor API differences between different versions (e.g. make all versions of Preact expose the same render or createRoot API).

Some possible versions to support for the libraries:

• local
• prev-local (for dev'ing and comparing your previous local change to the newest local change)
• master
• prev major
• prev minor (or whatever base to use for long term trend tracking)
• latest (i.e. current major, current published)
• next major

[andrewiggins]

Command line interface

Specifying the combinations of what to run will be determined by two command line parameters to the benchmark script:

• Dependency Group (-d): A comma separated list of versions of dependencies to run
• Implementations (--impl): Which implementations to run

Defaults

There are two levels of defaults:

• Globally defined default versions for each package
• Default implementations for a dependency version
  For example, the default implementation used for preact-v8 may be preact-class but for preact-v11 might be hooks. For signals dependency, the default implementation is the signals implementation. In a dependency group (e.g. preact/local,signals/local), the default implementation is determined by the first dependency listed

Examples

Here are some examples to illustrate usage. The ... represents the command to run the benchmark (perhaps npm run bench or something similar).

• Testing a core preact change
  ... table-app/replace1k.html -d preact/local -d preact/master

  Compare the performance of the preact/local vs preact/master on the table-app/replace1k.html benchmark using the default implementation for local and master

• Testing a hooks change
  ... table-app/replace1k.html -d preact/local -d preact/master --impl preact-hooks

  Compare the performance of preact/local vs preact/master on the table-app/replace1k.html benchmark using the preact-hooks implementations.
  Testing a compat change would work similarly, just change the --impl to preact-compat.

• Testing a core preact change on compat
  ... table-app/replace1k.html -d preact/local -d preact/master --impl preact-hooks

  Compare the performance of preact/local vs preact/master on the table-app/replace1k.html benchmark using the preact-hooks implementations.
  This command is the same as "Testing a hooks change" since hooks are sub-package of the core preact package.

• Testing a signals change
  ... table-app/replace1k.html -d signals/local -d signals/master

  Compare the performance of the signals/local vs signals/master package on the table-app/replace1k.html benchmark using the default implementation for signals (i.e. the signals implementation)

• Testing a core preact change on signals
  ... table-app/replace1k.html -d preact/local -d preact/master --impl preact-signals

  Compare the performance of preact/local vs preact/master on the table-app/replace1k.html benchmark using the signals implementation. This command would use the default version of the signals package

• Testing a core preact change with local signals implementation
  ... table-app/replace1k.html -d preact/local,signals/local -d preact/master,signals/local --impl preact-signals

  Compare the performance of preact/local + signals/local vs preact/master + signals/local on the table-app/replace1k.html benchmark using the signals implementation.

  If you swapped the order of the packages in the dependency groups (i.e. preact/local,signals/local -> signals/local,preact/local), then you could drop the --impl signals flag since it would default to the signals implementation for each dependency group (but explicitly defining the implementation you want to run is probably better)

• Comparing performance between major versions of preact
  ... table-app/replace1k.html -d preact/v10 -d preact/v11

  Compare the performance of preact/v8 vs preact/v10 on the table-app/replace10k.html benchmark using the default implementations for preact/v10 (e.g. preact-class) and preact/v11 (e.g. preact-hooks? or something else)

• Testing different implementations against each other
  ... table-app/replace1k.html --impl preact-hooks --impl preact-compat

  Compare the performance of the hooks implementation vs the compat implementation of the table-app/replace1k.html benchmark using global default versions for all dependencies.

• Testing different implementations against each other using local versions
  ... table-app/replace1k.html -d preact/local,signals/local --impl hooks --impl preact-signals

  Compare the performance of the hooks implementation vs the compat implementation of the table-app/replace1k.html benchmark using local versions for the preact and signals dependencies.

One benefit of this way of specifying what to run is its flexibility. However it does leave room for meaningless definitions. For example: ... table-app/replace1k.html -d signals/local -d signals/master --impl preact-hooks would be a meaningless run because the preact-hooks implementation doesn't use signals.

[andrewiggins]

Also consider the ability to do things like measure how a change in compat affects client and server rendering perf

[andrewiggins]

Would also be nice to have dev mode that caches the current code as "previous" and compares it against the new "current" to understand if a particular change impacted performance at all.