End-to-end testing

[echeran]

PR #86 has started thinking of designing end-to-end testing for components. Although it uses component structs to do comparisons on component API output, it specifies the data for test inputs and outputs using a custom set of structs (serializable to JSON).

This comment in the PR starts the discussion of how we can instead go through the data provider interface to fetch test data (inputs / outputs).

[sffc]

We could have an IterableDataProvider that gives hunks such as the following:

{
  input: "Zh-HANT",
  output: "zh-Hant"
}

This would be associated with a specific data key such as "testing/locale-canonic", or perhaps "locale/test-canonic".

We would dump this to a golden data file, and be able to re-generate the golden data file from actual behavior. Then for testing, we test that the raw golden data is equal to the re-generated golden data.

The golden data could have the following structure, e.g., with the FsDataProvider:

• testing
  • locale-canonic
    • test001.json
    • test002.json
  • locale-test2
    • test001.json
    • test002.json

But then you would also have GoldenDataGenerationProvider, which would re-generate the above structure from the actual icu::Locale class. GoldenDataGenerationProvider may depend on FsDataProvider to read the inputs, but instead of forwarding the same outputs, it would re-generate the outputs from ICU4X library code. Then, for testing, you would compare the output of GoldenDataGenerationProvider to FsDataProvider.

[echeran]

One thought that I had about the idea of using the data provider to provide test data, in a way that might be extensible to users, is that it seems related to #150 on "how do we hook a data provider to an API?". I think end-to-end testing could be another specific example for that question. And it seems to add evidence to our current working decision of having the data provider be an argument to low level APIs.

I'm still fuzzy on how that would look to make the data provider itself able to be swapped out by the user. It would need to be configured in some type of data somehow (ex: string identifier) within the test data so that the user can configure to use their own personal data provider. But there also has to be a mechanism (another interface / trait?) within the testing code that describes how to turn the data representing a data provider (ex: string identifier) and return the corresponding instance.

[sffc]

Merging this into #51 as a duplicate.