Skip to content

Commit

Permalink
[meta] update documentation for glob matcher + changelogs
Browse files Browse the repository at this point in the history
  • Loading branch information
sunshowers committed Dec 3, 2023
1 parent 0758503 commit 21e4f46
Show file tree
Hide file tree
Showing 6 changed files with 99 additions and 18 deletions.
10 changes: 10 additions & 0 deletions nextest-filtering/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,14 @@
# Changelog

## [0.6.0] - 2023-12-03

### Added

- Support for glob matchers.
- Support for a new `binary_id` predicate.

For more information, see the changelog for [cargo-nextest 0.9.64](https://nexte.st/CHANGELOG.html#0964---2023-12-03).

## [0.5.1] - 2023-10-22

### Changed
Expand Down Expand Up @@ -68,6 +77,7 @@

Initial release.

[0.6.0]: https://github.com/nextest-rs/nextest/releases/tag/nextest-filtering-0.6.0
[0.5.1]: https://github.com/nextest-rs/nextest/releases/tag/nextest-filtering-0.5.1
[0.5.0]: https://github.com/nextest-rs/nextest/releases/tag/nextest-filtering-0.5.0
[0.4.0]: https://github.com/nextest-rs/nextest/releases/tag/nextest-filtering-0.4.0
Expand Down
9 changes: 7 additions & 2 deletions nextest-metadata/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
# Changelog

## Unreleased
## [0.9.3] - 2023-12-03

### Added

- New method `RustBinaryId::from_parts` to construct a `RustBinaryId` from its constituent parts.
- Stabilized and documented [the binary ID format](https://docs.rs/nextest-metadata/latest/nextest_metadata/struct.RustBinaryId.html#method.from_parts).

## [0.9.2] - 2023-09-26

Expand Down Expand Up @@ -40,7 +45,6 @@

(This was not released due to a publishing issue.)


## [0.8.0] - 2023-01-02

### Added
Expand Down Expand Up @@ -183,6 +187,7 @@ This release is compatible with cargo-nextest 0.9.10 and later.

- Initial version, with support for listing tests.

[0.9.3]: https://github.com/nextest-rs/nextest/releases/tag/nextest-metadata-0.9.3
[0.9.2]: https://github.com/nextest-rs/nextest/releases/tag/nextest-metadata-0.9.2
[0.9.1]: https://github.com/nextest-rs/nextest/releases/tag/nextest-metadata-0.9.1
[0.9.0]: https://github.com/nextest-rs/nextest/releases/tag/nextest-metadata-0.9.0
Expand Down
5 changes: 5 additions & 0 deletions nextest-runner/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Changelog

## [0.51.0] - 2023-12-03

See the changelog for [cargo-nextest 0.9.64](https://nexte.st/CHANGELOG.html#0964---2023-12-03).

## [0.50.0] - 2023-11-16

See the changelog for [cargo-nextest 0.9.63](https://nexte.st/CHANGELOG.html#0963---2023-11-17).
Expand Down Expand Up @@ -400,6 +404,7 @@ Thanks to [Guiguiprim](https://github.com/Guiguiprim) for their contributions to

- Initial version.

[0.51.0]: https://github.com/nextest-rs/nextest/releases/tag/nextest-runner-0.51.0
[0.50.0]: https://github.com/nextest-rs/nextest/releases/tag/nextest-runner-0.50.0
[0.49.0]: https://github.com/nextest-rs/nextest/releases/tag/nextest-runner-0.49.0
[0.48.0]: https://github.com/nextest-rs/nextest/releases/tag/nextest-runner-0.48.0
Expand Down
24 changes: 24 additions & 0 deletions site/src/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,29 @@
This page documents new features and bugfixes for cargo-nextest. Please see the [stability
policy](book/stability.md) for how versioning works with cargo-nextest.

## [0.9.64] - 2023-12-03

### Added

- Stabilized and documented [the binary ID format](https://nexte.st/book/running#binary-ids).
- Support for glob matchers in [filter expressions](https://nexte.st/book/filter-expressions). For example, `package(foo*)` will match all tests whose names start with `foo`.
- A new `binary_id()` predicate matches against the binary ID.

### Changed

- Unit tests in proc-macro crates now have a binary ID that consists of just the crate name, similar to unit tests in normal crates.

- The default string matcher for the following predicates has changed from _equality_ to _glob_:

- Package-related matchers: `package()`, `deps()`, and `rdeps()`
- Binary-related matchers: `binary()`

The new `binary_id()` predicate also uses the glob matcher by default.

### Fixed

- Fixed a regression with some Cargo nightly-only features: see [guppy-rs/guppy#174](https://github.com/guppy-rs/guppy/pull/174) for more details.

## [0.9.63] - 2023-11-17

### Fixed
Expand Down Expand Up @@ -852,6 +875,7 @@ Supported in this initial release:
- [Test retries](book/retries.md) and flaky test detection
- [JUnit support](book/junit.md) for integration with other test tooling

[0.9.64]: https://github.com/nextest-rs/nextest/releases/tag/cargo-nextest-0.9.64
[0.9.63]: https://github.com/nextest-rs/nextest/releases/tag/cargo-nextest-0.9.63
[0.9.62]: https://github.com/nextest-rs/nextest/releases/tag/cargo-nextest-0.9.62
[0.9.61]: https://github.com/nextest-rs/nextest/releases/tag/cargo-nextest-0.9.61
Expand Down
50 changes: 42 additions & 8 deletions site/src/book/filter-expressions.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ cargo nextest run -E 'test(my_test) + package(my-crate)'
### Examples

- `package(serde) and test(deserialize)`: every test containing the string `deserialize` in the package `serde`
- `deps(nextest*)`: all tests in packages whose names start with `nextest`, and all of their (possibly transitive) dependencies
- `not (test(/parse[0-9]*/) | test(run))`: every test name not matching the regex `parse[0-9]*` or the substring `run`

> **Note:** If you pass in both a filter expression and a standard, substring-based filter, tests
Expand Down Expand Up @@ -62,6 +63,7 @@ This section contains the full set of operators supported by the DSL.
- `binary(name-matcher)`: include all tests in binary names matching `name-matcher`.
- For tests of kind `lib` and `proc-macro`, the binary name is the same as the name of the crate.
- Otherwise, it's the name of the integration test, benchmark, or binary target.
- `binary_id(name-matcher)`: include all tests in [binary IDs](running.md#binary-ids) matching `name-matcher`.
- `platform(host)` or `platform(target)`: include all tests that are [built for the host or target platform](running.md#filtering-by-build-platform), respectively.
- `none()`: include no tests.

Expand All @@ -79,18 +81,22 @@ This section contains the full set of operators supported by the DSL.
### Name matchers

- `~string`: match a package or test name containing `string`
- `=string`: match a package or test name that's equal to `string`
- `/regex/`: match a package or test name if any part of it matches the regular expression `regex`. To match the entire string against a regular expression, use `/^regex$/`. The implementation uses the [regex](https://github.com/rust-lang/regex) crate.
- `string`: default matching strategy.
- For tests (`test()`), this is equivalent to `~string`.
- For packages (`package()`, `deps()` and `rdeps()`), binary kinds (`kind()`), and `platform()`, this is equivalent to `=string`.
- `=string`: *equality matcher*—match a package or test name that's equal to `string`.
- `~string`: *contains matcher*—match a package or test name containing `string`.
- `/regex/`: *regex matcher*—match a package or test name if any part of it matches the regular expression `regex`. To match the entire string against a regular expression, use `/^regex$/`. The implementation uses the [regex](https://github.com/rust-lang/regex) crate.
- `#glob`: *glob matcher*—match a package or test name if the full name matches the glob expression `glob`. The implementation uses the [globset crate](https://docs.rs/globset).
- `string`: Default matching strategy.
- For `test()` predicates, this is the *contains matcher*, equivalent to `~string`.
- For package-related predicates (`package()`, `deps()`, and `rdeps()`), this is the *glob matcher*, equivalent to `#string`.
- For binary-related predicates (`binary()` and `binary_id()`), this is also the *glob matcher*.
- For `kind()` and `platform()`, this is the *equality matcher*, equivalent to `=string`.

If you're constructing an expression string programmatically, it is recommended that you always use a prefix to avoid ambiguity.
If you're constructing an expression string programmatically, **always use a prefix** to avoid ambiguity.

#### Escape sequences

The `~string` and `=string` name matchers can contain escape sequences, preceded by a backslash (`\`).
The *equality*, *contains*, and *glob* matchers can contain escape sequences, preceded by a
backslash (`\`).

- `\n`: line feed
- `\r`: carriage return
Expand All @@ -101,6 +107,8 @@ The `~string` and `=string` name matchers can contain escape sequences, preceded
- `\,`: comma
- `\u{7FFF}`: 24-bit Unicode character code (up to 6 hex digits)

For the *glob matcher*, to match against a literal glob metacharacter such as `*` or `?`, enclose it in square brackets: `[*]` or `[?]`.

All other escape sequences are invalid.

The _regular expression_ matcher supports the same escape sequences that [the regex crate does](https://docs.rs/regex/latest/regex/#escape-sequences). This includes character classes like `\d`. Additionally, `\/` is interpreted as an escaped `/`.
Expand Down Expand Up @@ -130,3 +138,29 @@ Within a precedence group, operators bind from left to right.
- `test(a) | test(b) & test(c)` is equivalent to `test(a) | (test(b) & test(c))`.
- `test(a) & test(b) - test(c)` is equivalent to `(test(a) & test(b)) - test(c)`.
- `not test(a) | test(b)` is equivalent to `(not test(a)) | test(b)`.

## More information about filter expressions

This section covers additional information that is of interest to nextest's developers and curious readers.

<details>
<summary>Click to expand</summary>

### Motivation

Developer tools often work with some notion of sets, and many of them have grown some kind of domain-specific query language to be able to efficiently specify those sets.

The biggest advantage of a query language is *orthogonality*: rather than every command having to grow a number of options such as `--include` and `--exclude`, developers can learn the query language once and use it everywhere.

### Design decisions

Nextest's filter expressions are meant to be specified at the command line as well as in configuration. This led to the following design decisions:

* **No quotes:** Filter expressions do not have embedded quotes. This lets users use either single (`''`) or double quotes (`""`) to specify filter expressions, without having to worry about escaping them.
* **Minimize nesting of parens:** If an expression language uses parentheses or other brackets heavily (e.g. Rust's [`cfg()` expressions]), getting them wrong can be annoying when trying to write an expression. Text editors typically highlight matching and missing parens, but there's so such immediate feedback on the command line.
* **Infix operators:** Nextest's filter expressions use infix operators, which are more natural to read and write for most people. (As an alternative, Rust's [`cfg()` expressions] use the prefix operators `all()` and `any()`).
* **Operator aliases:** Operators are supported as both words (`and`, `or`, `not`) and symbols (`&`, `|`, `+`, `-`, `!`), letting users write expressions in the style most natural to them. Filter expressions are a small language, so there's no need to be particularly opinionated.

[`cfg()` expressions]: https://doc.rust-lang.org/reference/conditional-compilation.html

</details>
19 changes: 11 additions & 8 deletions site/src/book/running.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,9 @@ In the output above:

- Tests are marked **`PASS`** or **`FAIL`**, and the amount of wall-clock time each test takes is listed within square brackets. In the example above, **`test_list_tests`** passed and took 0.052 seconds to execute.
- Tests that take more than a specified amount of time are marked **SLOW**. The timeout is 60 seconds by default, and can be [changed through configuration](configuration.md).
- The part of the test in purple is the _test binary_. A test binary is either:
- The part of the test in purple is the _binary ID_ for a unit test binary (see [Binary IDs](#binary-ids) below).

- a _unit test binary_ built from tests inline within `lib.rs`. These test binaries are shown by nextest as just the crate name, without a `::` separator inside them.
- an _integration test binary_ built from tests in the `[[test]]` section of `Cargo.toml` (typically tests in the `tests` directory.) These tests are shown by nextest in the format `crate-name::bin-name`[^bin-example].

For more about unit and integration tests, see [the documentation for `cargo test`](https://doc.rust-lang.org/cargo/commands/cargo-test.html).

- The part after the test binary is the _test name_, including the module the test is in. The final part of the test name is highlighted in bold blue text.
- The part after the binary ID is the _test name_, including the module the test is in. The final part of the test name is highlighted in bold blue text.

`cargo nextest run` supports all the options that `cargo test` does. For example, to only execute tests for a package called `my-package`:

Expand All @@ -31,7 +26,15 @@ cargo nextest run -p my-package

For a full list of options accepted by `cargo nextest run`, see `cargo nextest run --help`.

[^bin-example]: `bin` and `example` targets can also contain tests. Those are represented as `crate-name::bin/bin-name` and `crate-name::example/example-name`, respectively.
## Binary IDs

A test binary can be any of:

- A _unit test binary_ built from tests within `lib.rs` or its submodules. The binary ID for these are shown by nextest as just the crate name, without a `::` separator inside them.
- An _integration test binary_ built from tests in the `[[test]]` section of `Cargo.toml` (typically tests in the `tests` directory.) The binary ID for these is has the format `crate-name::bin-name`.
- Some other kind of test binary, such as a benchmark. In this case, the binary ID is `crate-name::kind/bin-name`. For example, `nextest-runner::bench/my-bench` or `quick-junit::example/show-junit`.

For more about unit and integration tests, see [the documentation for `cargo test`](https://doc.rust-lang.org/cargo/commands/cargo-test.html).

## Filtering tests

Expand Down

0 comments on commit 21e4f46

Please sign in to comment.