Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Detect code blocks more leniently #155

Closed
wants to merge 3 commits into from
Closed

Detect code blocks more leniently #155

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e3855e3
Detect code blocks more leniently
SomeoneToIgnore Jul 22, 2024
69ebd97
Fix the tests
SomeoneToIgnore Jul 22, 2024
4cfa6aa
Merge branch 'split_parser' into code-blocks-and-whitespaces
maxbrunsfeld Sep 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion tree-sitter-markdown/src/scanner.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -415,7 +415,7 @@ static bool parse_fenced_code_block(Scanner *s, const char delimiter,
if ((delimiter == '`' ? valid_symbols[FENCED_CODE_BLOCK_END_BACKTICK]
: valid_symbols[FENCED_CODE_BLOCK_END_TILDE]) &&
s->indentation < 4 && level >= s->fenced_code_block_delimiter_length &&
(lexer->lookahead == '\n' || lexer->lookahead == '\r')) {
iswspace(lexer->lookahead)) {
s->fenced_code_block_delimiter_length = 0;
lexer->result_symbol = delimiter == '`' ? FENCED_CODE_BLOCK_END_BACKTICK
: FENCED_CODE_BLOCK_END_TILDE;
Expand Down
6 changes: 4 additions & 2 deletions tree-sitter-markdown/test/corpus/spec.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1988,8 +1988,10 @@ Example 117 - https://github.github.com/gfm/#example-117
(fenced_code_block
(fenced_code_block_delimiter)
(block_continuation)
(code_fence_content
(block_continuation))
(ERROR
Copy link
Collaborator

@clason clason Jul 27, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this ERROR expected? Shouldn't this be parsed as a fenced_code_block containing "``` aaa"?

Copy link
Author

@SomeoneToIgnore SomeoneToIgnore Jul 27, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no idea what this grammars expects.
GitHub seems to render the way you've mentioned indeed: https://gist.github.com/SomeoneToIgnore/6c87cec69e6aa29fac76cb8ac7230912

Do you have any ideas how to fix it properly?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, sorry. I'd suspect the scanner needs to lookahead further to the end of the line, and then check if there are any non-whitespace characters.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What complicates things (besides my diminishing knowledge of the area) is the fact that pure split_parser branch fails for me in many tests:

  extension_wikilink:
    ✗ Basic Wiki-link parsing.
    ✗ Wiki-link to a file
    ✗ Wiki-link to a heading in a note
    ✗ Wiki-link with title
    ✓ Wiki-link opener with no closer
    ✗ Wiki-link version of Example 556
  failing:
     Example 355 - https://github.github.com/gfm/#example-355
     Example 363 - https://github.github.com/gfm/#example-363
     Example 389 - https://github.github.com/gfm/#example-389
     Example 394 - https://github.github.com/gfm/#example-394
     Example 400 - https://github.github.com/gfm/#example-400
     Example 401 - https://github.github.com/gfm/#example-401
     Example 407 - https://github.github.com/gfm/#example-407
     Example 406 - https://github.github.com/gfm/#example-406
     Example 411 - https://github.github.com/gfm/#example-411
     Example 420 - https://github.github.com/gfm/#example-420
     Example 421 - https://github.github.com/gfm/#example-421
     Example 424 - https://github.github.com/gfm/#example-424
     Example 438 - https://github.github.com/gfm/#example-438
     Example 524 - https://github.github.com/gfm/#example-524
     Example 538 - https://github.github.com/gfm/#example-538
     Example 560 - https://github.github.com/gfm/#example-560
     Example 588 - https://github.github.com/gfm/#example-588
     Example 635 - https://github.github.com/gfm/#example-635

which does not simplify things.
So I'd love some guidance.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry that I did not answer in time. As clason mentioned, the parser should not output an error here. The problem is, that you do not "consume" the trailing whitespaces. There needs to be another while loop like the one above, but with whitespace instead of delimiter. (If you still want to work on this.)

I'm not sure why the tests fail for you. Try using the instructions here if you didn't already.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I didn't see there was another PR that fixed this problem already.

(fenced_code_block_delimiter))
(code_fence_content))
(fenced_code_block
(fenced_code_block_delimiter))))

================================================================================
Expand Down