Data-driven testing

[echeran]

One part of the larger testing strategy for ICU4X would be to have a clean, consistent way of organizing the unit tests for the business logic (i18n algorithms). In particular, it would be nice to have a data-oriented style of testing, as exemplified already in some parts of @zbraniecki's unic-locale repo. ICU unit tests tend to be written in a parameterized style, but the idea here is to take the data-driven nature further.

Pros and Cons

Pros:

• ICU algorithms are mostly (all?) stateless, so describing in terms of data condenses test code to essential inputs and outputs
• Using a data representation makes tests language agnostic and reusable in target languages
• Representing tests as data would mirror the essence of user inputs and outputs described in the wrapping layer
• A concise notation would incentivize uniformity and concision of testing

Cons:

• Need to create a test harness/mini-framework to parse and execute test cases written as data
• Target languages wanting to reuse tests-as-data on ICU4X client libraries would need to write a native harness/mini-framework
• Tests that depend on stateful resources (ex: data stores) with setup/teardown phases would have to be handled in some other way
• Representing tests in a new way would require some initial adjustment

Existing libraries

Most searches for "data driven testing" produce results for databases, spreadsheets, and automated web UI testing. Links to more relevant pre-existing libraries are welcome.

Some examples of test libraries written to reduce the cognitive load when testing, especially when testing data collections:

• Truth - Fluent assertions for Java and Android
• Expectations - error messages that do a nested data diff on expected vs. actual

Testing features

Beyond just asserting that the actual return value matches the provided expected value, we should also consider the following testing aspects:

• matching functions/operators (equal, not equal, contains, does not contain, equal contents in {any, same} order, ...)
• test name
• test type (corresponding to ICU4X feature)
• message on error
• expected failure modes (ex: exceptions/panics)
• parameterization (single test input vs. list of related test inputs)
• APIs to read/construct test cases (ex: to enable interop with fuzz testing?)

[nciric]

Since we play a lot with JSON (test data etc) - this may be helpful - JSON Schema.

JSON Schema

• Describes your existing data format(s).
• Provides clear human- and machine- readable documentation.
• Validates data which is useful for:
  • Automated testing.
  • Ensuring quality of client submitted data.