Description
One part of the larger testing strategy for ICU4X would be to have a clean, consistent way of organizing the unit tests for the business logic (i18n algorithms). In particular, it would be nice to have a data-oriented style of testing, as exemplified already in some parts of @zbraniecki's unic-locale repo. ICU unit tests tend to be written in a parameterized style, but the idea here is to take the data-driven nature further.
Pros and Cons
Pros:
- ICU algorithms are mostly (all?) stateless, so describing in terms of data condenses test code to essential inputs and outputs
- Using a data representation makes tests language agnostic and reusable in target languages
- Representing tests as data would mirror the essence of user inputs and outputs described in the wrapping layer
- A concise notation would incentivize uniformity and concision of testing
Cons:
- Need to create a test harness/mini-framework to parse and execute test cases written as data
- Target languages wanting to reuse tests-as-data on ICU4X client libraries would need to write a native harness/mini-framework
- Tests that depend on stateful resources (ex: data stores) with setup/teardown phases would have to be handled in some other way
- Representing tests in a new way would require some initial adjustment
Existing libraries
Most searches for "data driven testing" produce results for databases, spreadsheets, and automated web UI testing. Links to more relevant pre-existing libraries are welcome.
Some examples of test libraries written to reduce the cognitive load when testing, especially when testing data collections:
- Truth - Fluent assertions for Java and Android
- Expectations - error messages that do a nested data diff on expected vs. actual
Testing features
Beyond just asserting that the actual return value matches the provided expected value, we should also consider the following testing aspects:
- matching functions/operators (equal, not equal, contains, does not contain, equal contents in {any, same} order, ...)
- test name
- test type (corresponding to ICU4X feature)
- message on error
- expected failure modes (ex: exceptions/panics)
- parameterization (single test input vs. list of related test inputs)
- APIs to read/construct test cases (ex: to enable interop with fuzz testing?)