Data-driven testing

[echeran]

One part of the larger testing strategy for ICU4X would be to have a clean, consistent way of organizing the unit tests for the business logic (i18n algorithms). In particular, it would be nice to have a data-oriented style of testing, as exemplified already in some parts of @zbraniecki's unic-locale repo. ICU unit tests tend to be written in a parameterized style, but the idea here is to take the data-driven nature further.

Pros and Cons

Pros:

• ICU algorithms are mostly (all?) stateless, so describing in terms of data condenses test code to essential inputs and outputs
• Using a data representation makes tests language agnostic and reusable in target languages
• Representing tests as data would mirror the essence of user inputs and outputs described in the wrapping layer
• A concise notation would incentivize uniformity and concision of testing

Cons:

• Need to create a test harness/mini-framework to parse and execute test cases written as data
• Target languages wanting to reuse tests-as-data on ICU4X client libraries would need to write a native harness/mini-framework
• Tests that depend on stateful resources (ex: data stores) with setup/teardown phases would have to be handled in some other way
• Representing tests in a new way would require some initial adjustment

Existing libraries

Most searches for "data driven testing" produce results for databases, spreadsheets, and automated web UI testing. Links to more relevant pre-existing libraries are welcome.

Some examples of test libraries written to reduce the cognitive load when testing, especially when testing data collections:

• Truth - Fluent assertions for Java and Android
• Expectations - error messages that do a nested data diff on expected vs. actual

Testing features

Beyond just asserting that the actual return value matches the provided expected value, we should also consider the following testing aspects:

• matching functions/operators (equal, not equal, contains, does not contain, equal contents in {any, same} order, ...)
• test name
• test type (corresponding to ICU4X feature)
• message on error
• expected failure modes (ex: exceptions/panics)
• parameterization (single test input vs. list of related test inputs)
• APIs to read/construct test cases (ex: to enable interop with fuzz testing?)

[nciric]

Since we play a lot with JSON (test data etc) - this may be helpful - JSON Schema.

JSON Schema

• Describes your existing data format(s).
• Provides clear human- and machine- readable documentation.
• Validates data which is useful for:
  • Automated testing.
  • Ensuring quality of client submitted data.

[sffc]

I'm merging #223 and #410 into this issue. Some notes from those issues:

• We want the data to come through the data provider, but we also want data that lives close to the test. Can we maybe put invariant (und) data in the test file and load resources for other locales?
• Is there a way to make the test data extensible by users?

[sffc]

Related: https://unicode-org.atlassian.net/browse/CLDR-15111