Align parser testing with fluent-syntax in JS and Python

[zbraniecki]

We currently test the parser using separate methodology and fixtures from what JS and Python do.

I'd like to unify this for 0.2 to make it easier to maintain a single corpus of test fixtures and make it easier to keep parsers in sync.

[xmo-odoo]

How would it work, a language-agnostic test suite (similar to toml or docopt), possibly in the main repository, and then each implementation provides a basic test executable which can be fed whatever input data is relevant and whose output can be checked for correctness?

How would locale issues be handled?

[zbraniecki]

I think something like that. I don't have a very clear picture or experience with this. With the upcoming Fluent 0.6 release we'll have at least two parsers (both in JS) that aim to produce the same AST. Maybe we could get Rust implementation to produce the exact same AST as well? Then we could assume that FTL(input)<-->JSON(output) should be the same.

The next step would be to align errors.

How would locale issues be handled?

What do you mean by that?

[xmo-odoo]

I think something like that. I don't have a very clear picture or experience with this. With the upcoming Fluent 0.6 release we'll have at least two parsers (both in JS) that aim to produce the same AST. Maybe we could get Rust implementation to produce the exact same AST as well?

Given fluent ultimately generates test I was expecting the test binaries could just do the entire flow (flow template, data input, text output) without having to care for the internal implementation details. Constraining every language to use a JS-defined AST would be odd. Plus assuming e.g. the mature JS asts stores a lot of metadata (for error-reporting purposes) it would require that all implementations reach that level before they can pass the test suite, instead of being able to build it up afterwards.

Though maybe the test binary would also be able to convert its internal AST to a pseudo-AST expected by the test harness, which just happens to match the JS AST.

What do you mean by that?

Given the NUMBER and DATETIME functions these would need to be tested as well, and done so across multiple locales, but that assumes all implementations have a minimum set of compatible locales?

[JohnDoneth]

Correct me if I'm wrong as I may be misinterpreting, but wouldn't converting to an internal JSON-representation or JS-like AST incur some overhead? It seems we would be throwing away a lot of niceties of the Rust language that a JS AST can't provide like enum fields, and be trading a easier testing environment for a more difficult Rust development cycle.

I like @xmo-odoo ideas of test driven development. If fluent-rs tests given the same inputs and FTL files that JS is given, and produces the same outputs, I don't see why a larger intermediary code base is needed.

As for aligning errors, since the FTL file is the source of truth, we could simply rework the parser to report the errors just the same as the JS implementation does during parsing?

[stasm]

This is a good point, thanks @JohnDoneth. I think the idea of being able to cross-test the AST output of the Rust parser with the output of the JS parsers comes from a larger plan to eventually replace the JS parsers with the Rust one. This requires some sort of alignment on the AST level to allow the JS resolver to format the translations parsed with the Rust parser.

I wonder if perhaps it would make sense to add a serialization layer on top of fluent-rs, not directly to it. I imagine a separate crate, maybe fluent-wasm, using the fluent-rs parser and this serialization layer to produce a JSON AST readily consumable by the JS runtime.

@zbraniecki I remember you tried using serde to serialize the Rust AST to JSON. Where did this effort end? Were you successful?

[zbraniecki]

I'm moving this to 0.4 for now, we separated out fluent-syntax crate that we'll likely update to syntax 0.7 or 0.8 and align with reference tests then.

[zbraniecki]

I now have a branch https://github.com/zbraniecki/fluent-rs/tree/copy-free which handles reference fixtures and is more or less in line with 0.7. I'll target that in 0.5 release.

[zbraniecki]

This has been fixed by #76