Description
@Centril and I talked a bit about writing property tests for CLI apps. The general idea is to write input generation and assertions on CLI app invocations without much overhead (e.g. without having to deal with manually formatting types as CLI arguments).
The most trivial implementation that already works is probably this:
proptest! {
#[test]
fn same_input_back(string: String) {
assert_cli::Assert::command(&["echo", &string])
.stdout().is(&string)
.unwrap();
}
}It generates random strings and asserts that echos gives us back the same string it was fed with.
We found that aside from the prop-testing specific challenges, there are also CLI-specific helpers needed, e.g. to easily generate temp dirs with (also generated) files, and that OS-specific or environment specific settings need to be able to be passed as arguments (or to be overwritten to reduce/eliminate dependence on global state).
In general, we think that we can use assert_cli to write these helpers. What we need to find is a way to express relations between arguments (inputs) and the asserts (like "stdout contains").
For the record but not meant as a concrete suggestions as something to implement today, I proposed a macro/DLS like:
cli_check!(cargo_binary("brighten"), ("--color": Color, "--amount": 0f32..=1) -> (stdout: Color));that passes two arguments to a binary called brighten and asserts that, given valid arguments, the tool writes a string to stdout that can be parsed as Color. Obviously missing here is a way to setup the environment. Side-effect of this syntax is that generating invalid (or omitting) arguments and asserting that the program fails is easy.