[Bug]: Tree-sitter cannot file when newline characters are missing

[pedeEli]

What happened?

A file that has carriage returns but no newline character can be parsed by R but not by tree-sitter.
Simple fix would be to insert a newline after every carriage return character if it does not have a newline after it already.

The file i used to figure this out was sunrise.set.R from the StreamMetabolism cran package.

What version of R do you have installed?

No response

What version of flowR did you use?

v2.2.8 (Feb 19, 2025)

Related input code

Relevant log output

Meta Information

• The Issue is Urgent
• I Used the Dockerized Version of flowr

[EagleoutIce]

@pedeEli i cannot reproduce using the parse utility of tree sitter. Can you please re-check?
Using the current main build, it parses fine for me without any additional handling necessary.

npm run flowr --silent -- --execute ":version; :parse file://<path>/StreamMetabolism/R/sunrise.set.R" --default-engine tree-sitter

[pedeEli]

sorry for the late answer. I tried running npm run flowr but i get an error. not sure why since i am on the latest main branch

> @eagleoutice/flowr@2.2.10 build:copy-wasm-min
> mkdir -p dist/src/cli && cp src/r-bridge/lang-4.x/tree-sitter/tree-sitter-r.wasm src/r-bridge/lang-4.x/tree-sitter/tree-sitter.wasm dist/src/cli

Syntaxfehler.

[EagleoutIce]

Do you have full log information for this?

[pedeEli]

D:\js\flowr> npm run flowr

> @eagleoutice/flowr@2.2.10 flowr
> npm run main --


> @eagleoutice/flowr@2.2.10 main
> npm run build:bundle-flowr && node dist/src/cli/flowr.min.js


> @eagleoutice/flowr@2.2.10 build:bundle-flowr
> npm run build && esbuild --bundle dist/src/cli/flowr.js --platform=node --tree-shaking=true --bundle --minify --external:clipboardy --target=node22 --outfile=dist/src/cli/flowr.min.js && npm run build:copy-wasm-min


> @eagleoutice/flowr@2.2.10 build
> tsc --project .


  dist\src\cli\flowr.min.js  1.1mb

Done in 1293ms

> @eagleoutice/flowr@2.2.10 build:copy-wasm-min
> mkdir -p dist/src/cli && cp src/r-bridge/lang-4.x/tree-sitter/tree-sitter-r.wasm src/r-bridge/lang-4.x/tree-sitter/tree-sitter.wasm dist/src/cli

Syntaxfehler.

[Ellpeck]

This looks to be an issue with the shell npm chose to run with. It defaults to using powershell on windows, which struggles with && and doesn't support multiple arguments for its version of cp :(

You can change the shell that npm uses to git bash through a global config though, and that should fix the build issue!

[pedeEli]

that worked. now to the original question. i tried running :parse file://path/to/file and it did work. i must have had an old version of tree-sitter, but the tree it generated is not correct:

R> :parse file://D:/Downloads/StreamMetabolism_1.1.3/StreamMetabolism/R/sunrise.set.R
program
╰ binary_operator
  ├ identifier "`sunrise.set`" (1:1─14)
  ├ <- "<-" (1:15─17)
  ╰ function_definition
    ├ function "function" (1:18─26)
    ├ parameters
    │ ├ ( "(" (1:26─27)
    │ ├ parameter
    │ │ ╰ identifier "lat" (1:27─30)
    │ ├ comma "," (1:30─31)
    │ ├ parameter
    │ │ ╰ identifier "long" (1:32─36)
    │ ├ comma "," (1:36─37)
    │ ├ parameter
    │ │ ╰ identifier "date" (1:38─42)
    │ ├ comma "," (1:42─43)
    │ ├ parameter
    │ │ ├ identifier "timezone" (1:44─52)
    │ │ ├ = "=" (1:52─53)
    │ │ ╰ string
    │ │   ├ " "\"" (1:53─54)
    │ │   ├ string_content "UTC" (1:54─57)
    │ │   ╰ " "\"" (1:57─58)
    │ ├ comma "," (1:58─59)
    │ ├ parameter
    │ │ ├ identifier "num.days" (1:64─72)
    │ │ ├ = "=" (1:72─73)
    │ │ ╰ float "1" (1:73─74)
    │ ╰ ) ")" (1:74─75)
    ╰ braced_expression
      ├ { "{" (1:75─76)
      ├ comment "#this needs to be long lat#\r      lat.long <- matrix(c(long, lat), nrow=1)\r      day <- as.POSIXct(date, tz=timezone)\r      sequence <- seq(from=day, length.out=num.days ,                   \t  by=\"days\")\r      sunrise <- sunriset(lat.long, sequence,  \t\t\t\t\t  direction=\"sunrise\",\r\t  POSIXct=TRUE)\r      sunset <- sunriset(lat.long, sequence,\t\t\t\t\t  direction=\"sunset\", POSIXct=TRUE)\r      ss <- data.frame(sunrise, sunset)\r      ss <- ss[,-c(1,3)]\r      colnames(ss)<-c(\"sunrise\", \"sunset\")\r      return(ss)\r}" (1:83─590)
      ╰ }  (1:590)

As you can see the last node is a comment and it contains program code since a comment is only stopped after a newline character. Technically this is the correct behaviour but it would be nice if flowr would parse it correctly.

[EagleoutIce]

Ok the parse tree being incorrect is another interesting problem that, for me, hints at a parsing error in the tree-sitter-r wasm (probably a comment searching for the next \n to terminate the line).

I propose a simple replacmenet of line endings as an intermediate solution. what do you say?

[pedeEli]

that should work yeah. i could also open an issue in the tree-sitter-r github. it would be nice if this issue is solved in the grammar itself

[EagleoutIce]

I agree, but even if this is an issue with the tree-sitter grammar I would suggest adding a compatibility option to the flowR config to allow for this patch. What do you say @Ellpeck?