Proper way to do C-Style comments

[dhowe]

I'm trying to implement single-line and multi-line C-style comments for my DSL. Is there an example I can look at. I basically want these elements to be skipped by the lexer, without messing up line/column numbers in errors. Thanks

[msujew]

I usually define C-Style comments like this:

const SingleLineComment = createToken({
  name: "SingleLineComment ",
  pattern: /\/\/[^\n\r]*/,
  group: Lexer.SKIPPED
})
const MultiLineComment = createToken({
  name: "MultiLineComment",
  pattern: /\/\*[\s\S]*?\*\//,
  group: Lexer.SKIPPED
})

Note the Lexer.SKIPPED group as outlined in this tutorial, that allows us to simply ignore comments during the parsing process.

[dhowe]

Thanks @msujew, single-line comments work for me only if I exclude the forward slash from other rules which may be longer.
For example, I have a token called RawText which matches most characters, including forward slash, and so, unless I disallow slashes in RawText (which I can't do), then comments don't work, as they are tokenized as RawText instead:

• <em>foo</em> -> [RawText]
• // hello -> [Comment]
• hello\n//hello -> [RawText, Comment]

[msujew]

I'm not sure I can follow. So what you're saying is that this string:

// hello
hello

is tokenized as [Comment, RawText]? That seems to make perfect sense to me, given that it's only a single line comment.

Anyway, I believe that you should be able to accomplish what you're looking for by ordering the token types correctly in your lexer and utilizing the longer_alt property.

[dhowe]

No, it is simply tokenized as [RawText]

But yes, I think I need the longer_alt property, just not sure how to use it

[msujew]

I see, then you just need to place SingleLineComment in front of RawText in the array that you provide to the lexer and set the longer_alt of SingleLineComment to RawText.

[dhowe]

Sorry, I mistyped the problem case above... It is:

hello
//hello

for which, no matter what I try, seems to always token only as RawText

[msujew]

I guess that makes sense. You probably need negative lookahead for // and /* in your RawText.

On another note, is there even a delimiter for RawText? It seems like it simply consumes everything once it starts, that could be a problem for your other tokens as well, not only comments.

[bd82]

Yeah this is generally a bad practice to have such kind of token without delimiters...

[dhowe]

This is just a simple test case to demonstrate the problem. Is there an example of a chevrotain lexer that uses negative lookahead ?

[bd82]

Does the regexp parser used by chevrotain support negative lookahead?
Anyhow I am not familiar with such an example. but in the most general case you can use custom token patterns

• https://chevrotain.io/docs/features/custom_token_patterns.html

These will enable you to implement most kind of logic (as code) in your lexing flow.

[dhowe]

wasn't able to get it working with a custom token pattern, so implemented for now as a pre-lexing step...