Recipe: indentation-sensitive languages

[aabounegm]

This is a guide on how to use the new IndentationAwareTokenBuilder, added to Langium in eclipse-langium/langium#1578 (published in v3.2.0).

[github-actions]

PR Preview Action v1.4.6
🚀 Deployed preview to https://eclipse-langium.github.io/langium-previews/pr-previews/pr-246/
on branch previews at 2024-08-29 14:46 UTC

[Lotes]

Really nice recipe. I have some comments about the insights to the implementation. I think we should add links or detailed information how this token builder/lexer works.

[aabounegm]

Thanks for your comments! I will address them as soon as a decision is reached about eclipse-langium/langium#1608 to edit the last section as well

[aabounegm]

@Lotes Thanks for waiting so long, all comments should be addressed now.
Note that the last commit adds a subsection that depends on eclipse-langium/langium#1647 getting merged first

[Lotes]

Just some minor things and one big question. The most of my thoughts were already resolved. Thanks.

[Lotes]

The with capital T

[aabounegm]

It was intended as a continuation of the sentence before it, only interrupted by the code snippet. Not sure if it makes sense or if it counts as a separate sentence 🤔

[Lotes]

Then, I would suggest to add 3 dots at the end of the first phase and at the beginning of the second phrase.

[Lotes]

Question about indentation in the implementation: How do you distinguish between spaces and tabs?
This could be an interesting point of the configuration to show here. Maybe in an own section or as appendix.
How to align this with an editor-config, see https://editorconfig.org or other approaches?

I guess you choose only spaces or tabs for the WS token, right?

[aabounegm]

That's a good question!
I thought a lot about the best approach here, and in the end decided not to discriminate between them, which is the simpler way. Alternatives included allowing only one or the other through a config parameter, or treating a tab as n spaces (again, for a configurable n). I thought these 2 alternatives were a bit too strict (though that's how Python behaves, for example, by prohibiting mixing them), and I thought that ideally I could issue a warning, but I couldn't find a way to accept a token and still issue a warning/error.

[aabounegm]

Actually, now that I think about it, I could add some payload to the returned token and then in the lexer check for the payload and add to the errors array, but then there would still be no way of making it a warning rather than an error. Perhaps LexerResult should be augmented to allow warnings?

[Lotes]

I think resolution of my question would block this recipe. I mean we can still change something afterwards. Extending the LexerResult sounds too much for this change.

Another question could be: How to write an indention-aware formatter? Is it even applicable or doable? How is it done for Python?
We do not have to answer this now. I was just interested about some consequences or follow-up tasks.

[aabounegm]

I do not yet have experience writing formatters in Langium, but I don't see why it would be difficult to do. Generally, there are 2 approaches: formatting and pretty printing. One way to implement a formatter is to search for some (anti-)patterns in the code and issue TextEdits just for them. Pretty printers normally use the AST/CST (or some other intermediate representation) and transform them back into code, regardless of how it initially looked like before parsing. (or at least that's how I understand the difference between them)
Both approaches seem possible with the indentation-aware tokens, though the second one (pretty printer) is probably easier to implement, assuming we want the formatter to ensure consistent indentation characters and sizes.

For Python, one of the most popular formatter is black, and it uses the pretty printing approach. Not sure how other formatters handle inconsistent indentation, tbh.

[Lotes]

I have nothing more to add right now. Let's wait for a second opinion on your recipe :-) .

[Lotes]

Then, I would suggest to add 3 dots at the end of the first phase and at the beginning of the second phrase.

[Lotes]

I think resolution of my question would block this recipe. I mean we can still change something afterwards. Extending the LexerResult sounds too much for this change.

Another question could be: How to write an indention-aware formatter? Is it even applicable or doable? How is it done for Python?
We do not have to answer this now. I was just interested about some consequences or follow-up tasks.

[aabounegm]

@Lotes Langium v3.2 has already been published with the indentation-aware token builder, and questions about its usage already started coming in (eclipse-langium/langium#1696). Are we waiting for another review or can this recipe be merged?