Beman Test Standards

[bretbrownjr]

Motivation

Clearly we want good tests in all Beman libraries. Contributors could use more specifics about how to go about that.

Completion Criteria

This issue will be complete when:

• There is documentation in the Beman Standard describing testing requirements
• The exemplar project meets the documented requirements for interactive development workflows
• The exemplar project meets the documented requirements through automation in Continuous Integration

Guidance

Testing is more nuanced than many would expect. To start, it probably makes sense to target the baseline expectations around testing. That is, to provide tests suitable for regression testing. When a new contributor to a Beman library makes a change, they should have an answer to the question, "Did I break any functionality?" That implies that these tests should be deterministic, fast, and portable. They should be incorporated into Continuous Integration. It should be clear from the developer documentation of each repo how to perform the tests, though it is acceptable to reference more general Beman documentation instead of maintaining many different instances of equivalent documentation.

Mentor

@bretbrownjr is happy to discuss, review, and answer any questions.

[wusatosi]

Hey I would love to make contribution to this.

I am just a bit confused on what to start. I do not have a strong technical background to this so I would need more guidance/ mentorship.

This is a subject clearly needing future discussions.

Things I can think of to potential add to the standard:

1. All build parameters (e. g. BUILD_TESTING in exemplar) have appropriate CI testing to make sure projects are still able to configure, build, test and install with reasonable parameter variants.
2. All public interfacing methods have appropriate unit tests when possible.
3. Some kind of unit testing best practices suggestion, e. g. Tests need to be self explanatory.
4. Constexpr function should check if they can actually be evaluated at compile time by using static_assert if possible. (This is stricter than constexpr requirements as static_assert needs consteval).
5. Uniform testing commands when possible, standardize CMake parameters like BUILD_TESTING toggle.
6. Maybe a specification about the testing environment, needs to be tested across compile classes, platforms, C++ versions. Preferably in CI tagged to each commit.
7. Test coverage....? Maybe a recommend test coverage percentage?

This won't really result in any new changes to the exemplar repo besides the test coverage reporting.

Does this sound like a good direction?