Skip to content

Data-driven testing #51

Open
Open
@echeran

Description

@echeran

One part of the larger testing strategy for ICU4X would be to have a clean, consistent way of organizing the unit tests for the business logic (i18n algorithms). In particular, it would be nice to have a data-oriented style of testing, as exemplified already in some parts of @zbraniecki's unic-locale repo. ICU unit tests tend to be written in a parameterized style, but the idea here is to take the data-driven nature further.

Pros and Cons

Pros:

  • ICU algorithms are mostly (all?) stateless, so describing in terms of data condenses test code to essential inputs and outputs
  • Using a data representation makes tests language agnostic and reusable in target languages
  • Representing tests as data would mirror the essence of user inputs and outputs described in the wrapping layer
  • A concise notation would incentivize uniformity and concision of testing

Cons:

  • Need to create a test harness/mini-framework to parse and execute test cases written as data
  • Target languages wanting to reuse tests-as-data on ICU4X client libraries would need to write a native harness/mini-framework
  • Tests that depend on stateful resources (ex: data stores) with setup/teardown phases would have to be handled in some other way
  • Representing tests in a new way would require some initial adjustment

Existing libraries

Most searches for "data driven testing" produce results for databases, spreadsheets, and automated web UI testing. Links to more relevant pre-existing libraries are welcome.

Some examples of test libraries written to reduce the cognitive load when testing, especially when testing data collections:

  • Truth - Fluent assertions for Java and Android
  • Expectations - error messages that do a nested data diff on expected vs. actual

Testing features

Beyond just asserting that the actual return value matches the provided expected value, we should also consider the following testing aspects:

  • matching functions/operators (equal, not equal, contains, does not contain, equal contents in {any, same} order, ...)
  • test name
  • test type (corresponding to ICU4X feature)
  • message on error
  • expected failure modes (ex: exceptions/panics)
  • parameterization (single test input vs. list of related test inputs)
  • APIs to read/construct test cases (ex: to enable interop with fuzz testing?)

Metadata

Metadata

Assignees

Labels

A-designArea: Architecture or designC-test-infraComponent: Integration test infrastructureS-epicSize: Major project (create smaller child issues)T-coreType: Required functionality

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions