Add benchmark files and update documentation

Author: Kobzol

collector/compile-benchmarks/README.md

@@ -32,11 +32,12 @@ They mostly consist of real-world crates.
   crate.
 - **helloworld**: A trivial program. Gives a lower bound on compile time.
 - **html5ever-0.26.0**: An HTML parser. Stresses macro parsing code.
+- **html5ever-0.31.0**: An HTML parser. Stresses macro parsing code.
 - **hyper-0.14.18**: A fairly large crate. Utilizes async/await, and used by
   many Rust programs. The crate uses cargo features to enable large portions of its
   structure and is built with `--features=client,http1,http2,server,stream`.
-- **image-0.24.1**: Basic image processing functions and methods for 
-  converting to and from various image formats. Used often in graphics 
+- **image-0.24.1**: Basic image processing functions and methods for
+  converting to and from various image formats. Used often in graphics
   programming.
 - **libc-0.2.124**: An interface to `libc`. Contains many declarations of
   types, constants, and functions, but relatively little normal code. Stresses
@@ -78,7 +79,7 @@ compiler in interesting ways.
   caused [poor performance](https://github.com/rust-lang/rust/issues/32278) in
   the past.
 - **ctfe-stress-5**: A stress test for compile-time function evaluation.
-- **deeply-nested-multi**: A small program containing multiple examples 
+- **deeply-nested-multi**: A small program containing multiple examples
   ([one](https://github.com/rust-lang/rust/issues/38528),
   [two](https://github.com/rust-lang/rust/issues/72408),
   [three](https://github.com/rust-lang/rust/issues/75992))
@@ -100,7 +101,7 @@ compiler in interesting ways.
 - **many-assoc-items**: Contains a struct with many associated items, which
   caused [quadratic behavior](https://github.com/rust-lang/rust/issues/68957)
   in the past.
-- **match-stress**: Contains examples 
+- **match-stress**: Contains examples
   (one involving [a huge enum](https://github.com/rust-lang/rust/issues/7462),
   one involving
   [`exhaustive_patterns`](https://github.com/rust-lang/rust/pull/79394)) of
@@ -119,7 +120,7 @@ compiler in interesting ways.
 - **tt-muncher**: Calls a quadratic TT muncher macro (based on `quote::quote!`)
   with a long input, which stresses macro expansion.
 - **tuple-stress**: Contains a single array of 65,535 nested `(i32, (f64, f64,
-  f64))` tuples. The data was extracted and reduced from a [program dealing
+f64))` tuples. The data was extracted and reduced from a [program dealing
   with grid coordinates](https://github.com/urschrei/ostn15_phf) that was
   causing rustc to [run out of
   memory](https://github.com/rust-lang/rust/issues/36799).
@@ -196,25 +197,21 @@ Rust code being written today.
       `"touch_file"` entry.
     - See [`collector/src/benchmark/mod.rs`](https://github.com/rust-lang/rustc-perf/blob/12cb796f8a932a891b385ba23a36d78a2867ace1/collector/src/benchmark/mod.rs#L24-L27) for a complete reference.
   - Consider adding one or more `N-*.patch` files for the `IncrPatched`
-    scenario.
-    - If it's a primary benchmark, you should definitely do this.
-    - These usually consist of a patch that adds a single
-      `println!("testing");` statement somewhere.
-    - Creating the patch against what you've committed so far might be useful.
-      Use `git diff` from the repository root, or `git diff --relative` within
-      the benchmark directory. Copy the output into the `N-*.patch` file.
-    - Do a test run with an `IncrPatched` scenario to make sure the patch
-      applies correctly, e.g. `target/release/collector bench_local +nightly
-      --id Test --profiles=Check --scenarios=IncrPatched
-      --include=$NEW_BENCHMARK`
+    scenario. - If it's a primary benchmark, you should definitely do this. - These usually consist of a patch that adds a single
+    `println!("testing");` statement somewhere. - Creating the patch against what you've committed so far might be useful.
+    Use `git diff` from the repository root, or `git diff --relative` within
+    the benchmark directory. Copy the output into the `N-*.patch` file. - Do a test run with an `IncrPatched` scenario to make sure the patch
+    applies correctly, e.g. `target/release/collector bench_local +nightly
+--id Test --profiles=Check --scenarios=IncrPatched
+--include=$NEW_BENCHMARK`
   - Add the new entry to `collector/compile-benchmarks/README.md`.
   - Add a new licensing entry to `collector/compile-benchmarks/REUSE.toml` (see existing entries
-  for inspiration).
+    for inspiration).
     - If the benchmark is artificial, use the `MIT OR Apache-2.0` license and set Rust Project
-    developers as the copyright owners (see e.g. `await-call-tree` as an example).
+      developers as the copyright owners (see e.g. `await-call-tree` as an example).
     - If the benchmark is a third-party crate, make sure to use its license. Try to find the
-    copyright owner in the crate's `COPYRIGHT` or `README` files. If you cannot find it, consider
-    using the copyright owner `<crate-name> contributors`.
+      copyright owner in the crate's `COPYRIGHT` or `README` files. If you cannot find it, consider
+      using the copyright owner `<crate-name> contributors`.
   - `git add` the `Cargo.lock` file, if it's not already part of the
     benchmark's committed code.
     - If the benchmark has a `.gitignore` file that contains `Cargo.lock`,
@@ -262,8 +259,8 @@ Rust code being written today.
     of the two versions to make sure nothing outrageous has happened. Once the
     PR is merged, make sure it's running correctly.
   - Second, remove the old version of the benchmark.
-  Doing it in two steps ensures we have continuity of profiling coverage in the
-  case where something goes wrong.
+    Doing it in two steps ensures we have continuity of profiling coverage in the
+    case where something goes wrong.
 - Compare the benchmarking time of the two versions.
 - When adding the new version, for `perf-config.json` and the `N-*.patch`
   files, use the corresponding files for the old version as a starting point.
@@ -277,11 +274,13 @@ benchmarks in rustc-perf are verbatim copies of third-party crates. We
 periodically do a mass update of these benchmarks.
 
 Benefits of this approach:
+
 - We ensure we are measuring compilation of the crates most people are using.
   This is most relevant for popular crates.
 - We get coverage of newer language features.
 
 Costs of this approach:
+
 - It takes time and effort.
 - We lose some data continuity.
   - But the stable set of benchmarks used for the dashboard are not affected,
@@ -308,5 +307,6 @@ Costs of this approach:
   maintainers.
 
 History:
+
 - The first mass update of third-party crates occurred in [March/April
   2022](https://hackmd.io/d9uE7qgtTWKDLivy0uoVQw).

collector/compile-benchmarks/REUSE.toml

@@ -110,6 +110,11 @@ path = "html5ever-0.26.0/**"
 SPDX-FileCopyrightText = "The html5ever Project Developers"
 SPDX-License-Identifier = "MIT OR Apache-2.0"
 
+[[annotations]]
+path = "html5ever-0.31.0/**"
+SPDX-FileCopyrightText = "The html5ever Project Developers"
+SPDX-License-Identifier = "MIT OR Apache-2.0"
+
 [[annotations]]
 path = "hyper-0.14.18/**"
 SPDX-FileCopyrightText = "hyper contributors"

collector/compile-benchmarks/html5ever-0.31.0/0-println.patch

@@ -0,0 +1,12 @@
+diff --git a/src/util/str.rs b/src/util/str.rs
+index 2c0ec3e2..a78ff669 100644
+--- a/src/util/str.rs
++++ b/src/util/str.rs
+@@ -10,6 +10,7 @@
+ use std::fmt;
+ 
+ pub(crate) fn to_escaped_string<T: fmt::Debug>(x: &T) -> String {
++    println!("testing");
+     // FIXME: don't allocate twice
+     let string = format!("{x:?}");
+     string.chars().flat_map(|c| c.escape_default()).collect()

collector/compile-benchmarks/html5ever-0.31.0/Cargo.toml

@@ -86,3 +86,5 @@ version = "2.0.2"
 
 [features]
 trace_tokenizer = []
+
+[workspace]

collector/compile-benchmarks/html5ever-0.31.0/perf-config.json

@@ -1,4 +1,5 @@
 {
-  "artifact": "library",
-  "category": "primary"
+  "touch_file": "src/lib.rs",
+  "category": "primary",
+  "artifact": "library"
 }