We should always test the parse trees generated

[kaby76]

Perhaps this is better in the Discussions section, but I just don't understand why we aren't performing tree diffs on every parse, across all targets. It's not going to slow down the tests that much and it really will catch errors in Antlr, e.g., css3, mdn-at-counter-style.css, explained here.

[kaby76]

PHP grammar, Java target fails when printing out trees. #3017. This is exactly why we need to compute trees always.

[RossPatterson]

We should at least compare the trees from each target for every test, and ensure that all targets produce the same tree. We could do that without needing any specific effort in any of the grammars. Ideally, every test would have an expected parse tree, and we would include that in the comparison as well, ensuring that all targets produce the expected tree. Unfortunately, there are very few expected tree files in the grammars-v4 repo right now - I checked them all during my work to improve testing of parse tree files. The downside of this is that most changes to a grammar would cause tests to fail, until the expected tree was re-generated, manually verified, and committed.

We could arbitrarily assert that the current parse tree of every test that succeeds is by definition correct, and check that tree into the repo as the expected result. We might even be correct for well over 90% of the tests.

[kaby76]

We should at least compare the trees from each target for every test, and ensure that all targets produce the same tree. We could do that without needing any specific effort in any of the grammars. Ideally, every test would have an expected parse tree, and we would include that in the comparison as well, ensuring that all targets produce the expected tree. Unfortunately, there are very few expected tree files in the grammars-v4 repo right now - I checked them all during my work to improve testing of parse tree files. The downside of this is that most changes to a grammar would cause tests to fail, until the expected tree was re-generated, manually verified, and committed.

We could arbitrarily assert that the current parse tree of every test that succeeds is by definition correct, and check that tree into the repo as the expected result. We might even be correct for well over 90% of the tests.

I think the plan is to generate a tree for every parse and compare with the previous. This will be across targets which will likely result in new bugs being discovered. To generate the .tree files, use either the trgen-generated code to remaster the .tree files or create by hand. We'll need to consider what to do with people who are completely phobic about installing dotnet and trgen (works everywhere), and running the driver code for any one of the targets they want in order to remaster the .tree and .errors files.

[kaby76]

I'm starting to address this issue. The easy solution is to just check in all the .tree files. But, people will likely have a hard time remastering the files, and some will likely forget to do it. So, the idea to generate the files first with a "gold standard"--aka Java--then compare the trees across targets is probably the way to go. We'd might consider not checking in the .tree files at all, except that the tree might be valuable when comparing different versions of Antlr, which the desc.xml could specify.