Add a new grammar renderer

[ehuss]

This introduces a new grammar renderer. Instead of trying to write the grammar in markdown/html hybrid, this introduces a new syntax that is parsed by the mdbook-spec plugin. This grammar is then converted into markdown/html hybrid, and also to railroad diagrams.

There are a lot of changes here (and some can be split into separate PRs if desired). A general overview of what to see here:

• Grammar rules are now written inside a code block (instead of a blockquote). The syntax is pretty similar to the old syntax with various small changes. See the docs/grammar.md file for a complete description.
• There is now a summary chapter which shows the entire grammar all on one page.
• The grammar is parsed by mdbook-spec/src/grammar/parser.rs into an internal representation.
• The internal representation is converted to markdown in mdbook-spec/src/grammar/render_markdown.rs, and railroad diagrams in mdbook-spec/src/grammar/render_railroad.rs.
  • The railroad diagrams are generated using the railroad crate.
  • There is a toggle button the show/hide the railroad diagrams. It uses localstorage to keep that state sticky.
• The basic definitions and driver in the mdbook plugin is in mdbook-spec/src/grammar.rs. There are several pieces here:
  • The internal representation.
  • Code to load the grammar from the code blocks inside the chapters.
  • Some validation.
  • Code that will replace the code block with the rendered output.
  • Code for handling the summary chapter.
• All nonterminals are now linked to the rule definition.
• The text may now link to grammar rules by just putting them in brackets like [FunctionParameters]. Link definitions are automatically added to every page.
• Some rules were added or changed to accommodate the new renderer. I think all changes are put into separate commits to help with reviewing.
• Various misc fixes, see the individual commits.

I'd like to thank @lukaslueg for creating the railroad library which made this possible.

Closes #221
Closes #398
Closes #596
Closes #1513
Closes #1677

[lukaslueg]

railroad upstream here.

The railroad codebase hasn't seen a lot of love with respect to graphical layout. Suggestions are welcome.

AFAICS there are cases where the grammar diverges from its graphical representation with respect to repeated elements. In the two examples below, the diagram only allows for for at least two consecutive Statement (... two consecutive Expression), while the grammar requires "at least one".

[ia0]

The railroad codebase hasn't seen a lot of love with respect to graphical layout. Suggestions are welcome.

From the live demo I can see that *-repeated elements are in theory printed like this:

   .------->-------.
   |               |
->-+-+--[ foo ]--+-+--->-
     |           |
     '-----<-----'

What about doing it like this?

->-+------>------+->-
   |             |
   '-<-[ foo ]-<-'

This uses one less path and can be concatenated with a previous foo for +-repeat:

->-[ foo ]->-+------>------+->-
             |             |
             '-<-[ foo ]-<-'

The main problem is that foo is somehow to be read backwards, which may confuse people at first.

[lukaslueg]

The railroad codebase hasn't seen a lot of love with respect to graphical layout. Suggestions are welcome.

From the live demo I can see that *-repeated elements are in theory printed like this:

   .------->-------.
   |               |
->-+-+--[ foo ]--+-+--->-
     |           |
     '-----<-----'

What about doing it like this?

->-+------>------+->-
   |             |
   '-<-[ foo ]-<-'

This uses one less path and can be concatenated with a previous foo for +-repeat:

->-[ foo ]->-+------>------+->-
             |             |
             '-<-[ foo ]-<-'

The main problem is that foo is somehow to be read backwards, which may confuse people at first.

With respect to *-Elements, both examples (1 and 2) are technically valid. See the Zero or more table-constraints block in the create-table-stmt example, which demonstrates the second case.
Also see the One or more column-definitions-example, which should cover the +-Element case in example 3.

It's possible to implement dyn Node downstream to build more specialized primitives for certain situations. For instance, one might want to cook up a graphical representation for the "any character except ..."-case. Upstream might also provide them, if the need arises.

[ia0]

I see, so that's already supported and just a matter of generating the proper diagram downstream.

[ehuss]

Oops, thanks! I messed up all the repeats.

I'm a bit uncertain on how to handle the non-greedy ones, but I think a label works ok for now. Same with the a..b repeat range, and I went with a comment for that, which I think works ok?

[traviscross]

For instance, one might want to cook up a graphical representation for the "any character except ..."-case. Upstream might also provide them, if the need arises.

@lukaslueg: I'm curious if you have thoughts about what would make a good graphical representation for this. As I'm sure you saw, we use this pattern a lot, and I had found myself wondering about whether there might be a good visual way to encode this.

[lukaslueg]

Some comments and suggestions, which you hopefully find constructive

• A LabeledBox can use any primitive as the label, including a Sequence, just just plain text. See example 12_select-stmt_diagram.txt in railroad_dsl. This might be useful in situations where a Terminal or NonTerminal is referenced, as in "except \0 or \x00" and "except $ and delimiters" (where "delimiters" is a NonTerminal); yes, the NonTerminal in the Comment can have a clickable Link :-) railroad_dsl's syntax allows to access that, and this here might as well.
  
  
• Common pre- and suffixes in diverging arms (Choice) can be combined, shortening the Choice. One can use mathematical rigor to compute that, yet I found that real-world scenarios benefit from less strictness; there is a rather ridiculously complex implementation for this in macro_railroad here (unless you can control this manually here?!).
  
• There are some mistakes with respect to repeating elements in b2c1d6ada. Notice that reading direction reverses for the combining element (right to left, since we are going backwards within the syntax). For instance, the syntax for StructFields allows for "StructField StructField , ," (struct foobar { foo: i32 bar: i32, , }). Also notice that WhereClause allows for where , T: std::fmt::Debug U: std::fmt::Debug (both commas wrong). Unless you already did, you might want to have a look at the examples in railroad_dsl, which (afaics) cover those cases here (*/+/?).
  
  
• One can crate a dyn Node that wraps a LabeledBox, where the wrapping type pre-sets the label and adds another CSS-class in order to visualize non greedy regions (think "green box")
• With respect to "Anything except" (@traviscross) my first inclination would be a LabeledBox with custom CSS (think "red box"). Remember that the label can be a dyn Node itself, so the exceptions to the rule can at least have some unicode label (⚠) automatically attached to them.
• "Anything except" is actually just a case of "early parse error". There is no primitive for that, as we would also have to disambiguate diverging arms if anything in the diagram could lead to a parse error. AFAICS this is not a good solution.

[lukaslueg]

The problem with "except" is that its not immediately clear if the except-element is an immediate parse error or a delayed parse error, in case both are possible. E.g. if Foo X is invalid but Foo XY is not, the graphical (stateless!) representation becomes murky and hard to follow. Following the diagram does not require knowing where we are coming from // no memory.

As suggested above, imho a comment with NonTerminal/Terminal inside them - spiced up with some css - probably suffices, as there probably are cases which are either complex or verbose, and require clear text anyway ("except \u{0}, \u{00}, …, \u{000000}").

These should definitely get flipped around: Whatever "any character" is should be a NonTerminal/Terminal, and the exception (Terminal LF) should be part of the comment. If we had a dyn Node for "except" ("yellow box" - red is error) it would clearly indicate that there is an exception to what the NonTerminal can match.

[traviscross]

Yes, makes sense.

[lukaslueg]

See here for a quick and dirty example of how an except node might look like

[traviscross]

That's quite nice, yes. That way, the "except for" nodes aren't in the main railroad, which is what makes the other ways awkward.

[ehuss]

Thanks @lukaslueg! I went ahead and added your suggestion for the Except clauses.

To fix the repeat expressions, I had to essentially reverse the Sequence blocks inside a repeat. I couldn't find any other way to get around the HDir invert.

As for some of the other suggestions here, such as rendering nodes inside comments, those sound like good things to try in the future. That in particular might be challenging because the content is written in markdown.