Merge pull request #119 from sourcefrog/return-error

Optionally return errors from functions returning Result

.cargo/mutants.toml

@@ -1,3 +1,4 @@
 # cargo-mutants configuration
 
-exclude_globs = [ "src/console.rs" ]
+error_values = ["::anyhow::anyhow!(\"mutated\")"]
+exclude_globs = ["src/console.rs"]

.vscode/settings.json

@@ -0,0 +1 @@
+{}

Cargo.toml

@@ -89,6 +89,7 @@ exclude = [
     "testdata/tree/cfg_attr_mutants_skip",
     "testdata/tree/cfg_attr_test_skip",
     "testdata/tree/dependency",
+    "testdata/tree/error_value",
     "testdata/tree/everything_skipped",
     "testdata/tree/factorial",
     "testdata/tree/fails_without_feature",

NEWS.md

@@ -11,6 +11,11 @@
 
 - Minimum supported Rust version increased to 1.65 due to changes in dependencies.
 
+- New `--error` option, to cause functions returning `Result` to be mutated to return the
+  specified error.
+
+- New `--no-config` option, to disable reading `.cargo/mutants.toml`.
+
 ## 1.2.2
 
 Released 2023-04-01

book/src/SUMMARY.md

@@ -8,13 +8,15 @@
   - [Exit codes](exit-codes.md)
   - [The `mutants.out` directory](mutants-out.md)
 - [Skipping untestable code](skip.md)
+  - [Skipping functions with an attribute](attrs.md)
+  - [Filtering files](skip_files.md)
+  - [Filtering functions and mutants](filter_mutants.md)
 - [Controlling cargo-mutants](controlling.md)
   - [Listing and previewing mutations](list.md)
-  - [Filtering files](skip_files.md)
-  - [Filtering mutants](filter_mutants.md)
   - [Workspaces and packages](workspaces.md)
   - [Passing options to Cargo](cargo-args.md)
-  - [The `mutants.toml` config file](config.md)
+- [Generating mutants](mutants.md)
+  - [Error values](error-values.md)
 - [Improving performance](performance.md)
   - [Parallelism](parallelism.md)
 - [Integrations](integrations.md)

book/src/attrs.md

@@ -0,0 +1,51 @@
+# Skipping functions with an attribute
+
+To mark functions as skipped, so they are not mutated:
+
+1. Add a Cargo dependency on the [mutants](https://crates.io/crates/mutants)
+   crate, version "0.0.3" or later. (This must be a regular `dependency` not a
+   `dev-dependency`, because the annotation will be on non-test code.)
+
+2. Mark functions with `#[mutants::skip]` or other attributes containing
+   `mutants::skip` (e.g. `#[cfg_attr(test, mutants::skip)]`).
+
+The `mutants` create is tiny and the attribute has no effect on the compiled
+code. It only flags the function for cargo-mutants. However, you can avoid the
+dependency by using the slightly longer `#[cfg_attr(test, mutants::skip)]` form.
+
+**Note:** Currently, `cargo-mutants` does not (yet) evaluate attributes like
+`cfg_attr`, it only looks for the sequence `mutants::skip` in the attribute.
+
+You may want to also add a comment explaining why the function is skipped.
+
+For example:
+
+```rust
+use std::time::{Duration, Instant};
+
+/// Returns true if the program should stop
+#[cfg_attr(test, mutants::skip)] // Returning false would cause a hang
+fn should_stop() -> bool {
+    true
+}
+
+pub fn controlled_loop() {
+    let start = Instant::now();
+    for i in 0.. {
+        println!("{}", i);
+        if should_stop() {
+            break;
+        }
+        if start.elapsed() > Duration::from_secs(60 * 5) {
+            panic!("timed out");
+        }
+    }
+}
+
+mod test {
+    #[test]
+    fn controlled_loop_terminates() {
+        super::controlled_loop()
+    }
+}
+```

book/src/cargo-args.md

@@ -18,18 +18,28 @@ For example
 cargo mutants -- --cargo-arg=--release
 ```
 
+or in `.cargo/mutants.toml`:
+
+```toml
+additional_cargo_args = ["--all-features"]
+```
+
 ## Arguments to `cargo test`
 
 Command-line options following a `--` delimiter are passed through to
 `cargo test`. For example, this can be used to pass `--all-targets` which (unobviously)
 excludes doctests. (If the doctests are numerous and slow, and not relied upon to catch bugs, this can improve performance.)
 
-These options can also be configured statically with the `additional_cargo_test_args` key in `.cargo/mutants.toml`.
-
 ```shell
 cargo mutants -- --all-targets
 ```
 
+These options can also be configured statically with the `additional_cargo_test_args` key in `.cargo/mutants.toml`:
+
+```toml
+additional_cargo_test_args = ["--jobs=1"]
+```
+
 ## Arguments to test binaries
 
 You can use a second double-dash to pass options through to the test targets:

book/src/controlling.md

@@ -1,16 +1,30 @@
 # Controlling cargo-mutants
 
-`cargo mutants` takes various options to control how it runs. These options are shown in `cargo mutants --help` and are described in detail in this section.
+`cargo mutants` takes various options to control how it runs.
+
+These options, can, in general, be passed on the command line, set in a `.cargo/mutants.toml`
+file in the source tree, or passed in `CARGO_MUTANTS_` environment variables. Not every
+method of setting an option is available for every option, however, as some would not
+make sense or be useful.
+
+For options that take a list of values, values from the configuration file are appended
+to values from the command line.
+
+For options that take a single value, the value from the command line takes precedence.
+
+`--no-config` can be used to disable reading the configuration file.
+
+## Execution order
 
 By default, mutants are run in a randomized order, so as to surface results from
 different parts of the codebase earlier. This can be disabled with
-`--no-shuffle`, in which case mutants will run in the same order shown by
-`--list`: in order by file name and within each file in the order they appear in
+`--no-shuffle`, in which case mutants will run in order by file name and within each file in the order they appear in
 the source.
 
 ## Source directory location
 
-`-d`, `--dir`: Test the Rust tree in the given directory, rather than the default directory.
+`-d`, `--dir`: Test the Rust tree in the given directory, rather than the source tree
+enclosing the working directory where cargo-mutants is launched.
 
 ## Console output
 
@@ -20,10 +34,6 @@ the source.
 
 `--no-times`: Don't print elapsed times.
 
-## Environment variables
-
-A few options that may be useful to set globally can be configured through environment 
-variables:
-
-* `CARGO_MUTANTS_JOBS`
-* `CARGO_MUTANTS_TRACE_LEVEL`
+`-L`, `--level`, and `$CARGO_MUTANTS_TRACE_LEVEL`: set the verbosity of trace
+output to stdout. The default is `info`, and it can be increased to `debug` or
+`trace`.

book/src/error-values.md

@@ -0,0 +1,32 @@
+# Generating error values
+
+cargo-mutants can be configured to generate mutants that return an error value from functions that return a Result.
+
+This will flag cases where no test fails if the function returns an error: that might happen if there are _only_ tests for the error cases and not for the Ok case.
+
+Since crates can choose to use any type for their error values,
+cargo-mutants must be told how to construct an appropriate error.
+
+The `--error` command line option and the `error_value` configuration option specify an error value to use.
+
+These options can be repeated or combined, which might be useful
+if there are multiple error types in the crate. On any one mutation site, probably only one of the error values will be viable, and cargo-mutants will discover that and use it.
+
+The error value can be any Rust expression that evaluates to a value of the error type. It should not include the `Err` wrapper, because cargo-mutants will add that.
+
+For example, if your crate uses `anyhow::Error` as its error type, you might use `--error '::anyhow::anyhow!("error")'`.
+
+If you have your own error type, you might use `--error 'crate::MyError::Generic'`.
+
+Since the correct error type is a property of the source tree, the configuration should typically go into `.cargo/mutants.toml` rather than being specified on the command line:
+
+```toml
+error_values = ["::anyhow::anyhow!(\"mutated\")"]
+```
+
+To see only the mutants generated by this configuration, you
+can use a command like this:
+
+```sh
+cargo r mutants -F anyhow -vV -j4
+```

book/src/filter_mutants.md

@@ -1,23 +1,33 @@
 # Filtering functions and mutants
 
-The `#[mutants::skip]` attributes let you permanently mark a function as skipped
-in the source tree. You can also narrow down which functions or mutants are
-tested just for a single run of `cargo mutants`.
+You can also filter mutants by name, using the `--re` and `--exclude-re` command line
+options and the corresponding `examine_re` and `exclude_re` config file options.
 
-Two options filter mutants by the full name of the mutant, which includes the
-function name, file name, and a description of the change.
+These options are useful if you want to run cargo-mutants just once, focusing on a subset of functions or mutants.
 
-Mutant names are shown by `cargo mutants --list`, and the same command can be
-used to preview the effect of filters.
+These options filter mutants by the full name of the mutant, which includes the
+function name, file name, and a description of the change, as shown in list.
 
-`-F REGEX`, `--re REGEX`: Only test mutants whose full name matches the given regex.
+For example, one mutant name might be:
 
-`-E REGEX`, `--exclude-re REGEX`: Exclude mutants whose full name matches
-the given regex.
+```text
+src/outcome.rs:157: replace <impl Serialize for ScenarioOutcome>::serialize -> Result<S::Ok, S::Error> with Ok(Default::default())
+```
 
-These options may be repeated.
+Within this name, your regex can match any substring, including for example:
 
-The regex matches a substring and can be anchored with `^` and `$`.
+- The filename
+- The trait, `impl Serialize`
+- The struct name, `ScenarioOutcome`
+- The function name, `serialize`
+- The mutated return value, `with Ok(Defualt::default())`, or any part of it.
+
+Mutants can also be filtered by name in the `.cargo/mutants.toml` file, for example:
+
+Regexes from the config file are appended to regexes from the command line.
+
+The regex matches a substring, but can be anchored with `^` and `$` to require that
+it match the whole name.
 
 The regex syntax is defined by the [`regex`](https://docs.rs/regex/latest/regex/)
 crate.
@@ -32,3 +42,10 @@ Examples:
 
 - `-F 'impl Serialize' -F 'impl Deserialize'` -- test implementations of these
   two traits.
+
+Or in `.cargo/mutants.toml`:
+
+```toml
+exclude_re = ["impl Debug"] # same as -E
+examine_re = ["impl Serialize", "impl Deserialize"] # same as -F, test *only* matches
+```

book/src/mutants.md

@@ -0,0 +1,27 @@
+# Generating mutants
+
+cargo mutants generates mutants by inspecting the existing
+source code and applying a set of rules to generate new code
+that is likely to compile but have different behavior.
+
+In the current release, the only mutation pattern is to
+replace function bodies with a value of the same type.
+This checks that the tests:
+
+1. Observe any side effects of the original function.
+2. Distinguish return values.
+
+More mutation rules will be added in future releases.
+
+| Return type | Mutation pattern |
+| ----------- | ---------------- |
+| `()`        | `()` (return unit, with no side effects) |
+| `bool`      | `true`, `false` |
+| `String`    | `String::new()`, `"xyzzy".into()` |
+| `Result`    | `Ok(Default::default())`, [and an error if configured](error-values.md) |
+| (any other) | `Default::default()` (and hope the type implements `Default`) |
+
+Some of these values may not be valid for all types: for example, returning
+`Default::default()` will work for many types, but not all. In this case the
+mutant is said to be "unviable": by default these are counted but not printed,
+although they can be shown with `--unviable`.

book/src/parallelism.md

@@ -1,20 +1,30 @@
 # Parallelism
 
-The `--jobs` or `-j` option allows to test multiple mutants in parallel, by spawning several Cargo processes. This can give 25-50% performance improvements, depending on the tree under test and the hardware resources available.
+After the initial test of the unmutated tree, cargo-mutants can test multiple
+mutants in parallel. This can give significant performance improvements,
+depending on the tree under test and the hardware resources available.
 
-It's common that for some periods of its execution, a single Cargo build or test job can't use all the available CPU cores. Running multiple jobs in parallel makes use of resources that would otherwise be idle.
+Even though cargo builds, rustc, and Rust's test framework launch multiple
+processes or threads, they typically can't use all available CPU cores all the
+time, and many `cargo test` runs will end up using only one core waiting for the
+last task to complete. Running multiple jobs in parallel makes use of resources
+that would otherwise be idle.
 
-However, running many jobs simultaneously may also put high demands on the
-system's RAM (by running more compile/link/test tasks simultaneously), IO
-bandwidth, and cooling (by fully using all cores).
+By default, only one job is run at a time.
 
-The best setting will depend on many factors including the behavior of your program's test suite, the amount of memory on your system, and your system's behavior under high thermal load.
+To run more, use the `--jobs` or `-j` option, or set the `CARGO_MUTANTS_JOBS`
+environment variable.
 
-The default is currently to run only one job at a time. Setting this higher than the number of CPU cores is unlikely to be helpful.
+Setting this higher than the number of CPU cores is unlikely to be helpful.
 
-`-j 4` may be a good starting point, even if you have many more CPU cores. Start
-there and watch memory and CPU usage, and tune towards a setting where all cores
-are always utilized without memory usage going too high, and without thermal
-issues.
+The best setting will depend on many factors including the behavior of your
+program's test suite, the amount of memory on your system, and your system's
+behavior under high thermal load.
 
-Because tests may be slower with high parallelism, you may see some spurious timeouts, and you may need to set `--timeout` manually to allow enough safety margin.
+`-j 4` may be a good starting point. Start there and watch memory and CPU usage,
+and tune towards a setting where all cores are fully utilized without apparent
+thrashing, memory exhaustion, or thermal issues.
+
+Because tests may be slower with high parallelism, you may see some spurious
+timeouts, and you may need to set `--timeout` manually to allow enough safety
+margin.

book/src/performance.md

@@ -31,6 +31,7 @@ cargo-mutants causes the Rust toolchain (and, often, the program under test) to
 ```shell
 sudo mkdir /ram
 sudo mount -t tmpfs /ram /ram  # or put this in fstab, or just change /tmp
+sudo chmod 1777 /ram
 env TMPDIR=/ram cargo mutants
 ```