Add indentation-aware TokenBuilder and Lexer

[aabounegm]

This is useful for languages where indentation is used as a block delimiter, such as Python.
I will try to add a recipe for it in the documentation (and maybe some unit tests) later on.

The code was heavily inspired by the Python indentation chevrotain example, but modified to be more extensible and hopefully fitting for Langium.

Usage example (just a snippet, not full grammar):

fragment Body:
  INDENT
  statements+=Statement
  DEDENT
;

If: 'if' cond=Expr ':' Body;

hidden terminal NEW_LINE: /\r?\n/;
terminal INDENT: 'synthetic:indent'; // The content doesn't really matter.
terminal DEDENT: 'synthetic:dedent'; // It will be overridden anyway
hidden terminal WS: /[ \t]+/; // whitespace and newlines now have to be separated

The main drawback is that the whole language now becomes indentation-sensitive, even areas which would normally not be white-space sensitive, such as the following Python example:

numbers = [
  1,  // unexpected indentation
  2,
]

Multi-mode lexing can be leveraged to overcome this, and if I find a generic way to integrate it with the code from this PR, I will follow it up with another one that does so.

Closes #1016.
Related to #608, #663, #782, and #1085.

[aabounegm]

Whoops, looks like I messed up a bit with the imports there. That's what I get for using github.dev without cloning locally, I guess 😅. I'll fix it in a sec

[aabounegm]

@msujew Should hopefully be fine now 😄

[msujew]

Thanks for the contribution @aabounegm!

Do you mind creating some tests cases for the new lexer?

[aabounegm]

Sure, no problem, but that might take me some time (not sure if I will be able to manage it today)

[msujew]

@aabounegm All good, take your time. I appreciate it!

[spoenemann]

Thanks for the contribution! From a code organization standpoint, I'd prefer the two new service classes to be in a new file parser/indentation-aware.ts.

[aabounegm]

Thanks for your comments, I have now added unit tests and moved the new classes to a new file as suggested.
Please feel free to nitpick, or to edit directly if you so prefer

[msujew]

Thanks @aabounegm, looks pretty good to me 👍

[aabounegm]

Thank you for your fixes!

[spoenemann]

👍