Raw string literals (more than one #, grammar limitations prevent us from supporting escapes)

[RedCMD]

swift-tmlanguage/Swift.tmLanguage.yaml

Line 2339 in 4028d24

# SE-0168: raw string literals (more than one #, grammar limitations prevent us from supporting escapes)

What limitation are you facing?
syntax examples?

https://github.com/apple/swift-evolution/blob/main/proposals/0168-multi-line-string-literals.md
doesn't seem to say anything about hashtags #

is it this section?
https://github.com/apple/swift-evolution/blob/main/proposals/0200-raw-string-escaping.md#customized-escape-delimiters

[RedCMD]

I see the problem now

could possibly add a single line raw string literal that supports all the delimiters

but then you probably couldn't support \#( 'meta.embedded.line.swift' ) 😔

[RedCMD]

I think I've managed to get something to work
and is probably most cursed textmate I've ever written (worse than ((?!(?<!\\\\)\\\\{2}*\").)*?)

just had "comment" as a place holder to visualize whats being tokenized

it works by chaining begin/end groups one after another
the end rule doesn't capture anything. allowing the begin rule to retarget the escape delimiter and "literals-string-raw-string-guts" to tokenize

annoying I had to make an exception for the first set of non-escape-delimiters
(?>\\\\(?!\\1(?!#))#+)*
otherwise the very first \# and \### would be passed into "literals-string-raw-string-guts" and be marked as invalid (for \## string delimiters)

{
	"name": "swift-raw-strings",
	"scopeName": "source.redcmd.syntax.swift-raw-strings",
	"patterns": [ { "include": "#swift-raw-strings" } ],
	"repository": {
		"swift-raw-strings": {
			"begin": "(##+)\"",
			"end": "\"\\1(#*)",
			"beginCaptures": { "0": { "name": "punctuation.definition.string.begin.raw.swift" } },
			"endCaptures": {
				"0": { "name": "punctuation.definition.string.end.raw.swift" },
				"1": { "name": "invalid.illegal.extra-closing-delimiter.swift" }
			},
			"patterns": [
				{
					"begin": "(?<=(?>[^#]|^)(#+)\")\\G(?>\\\\(?!\\1(?!#))#+)*",
					"end": "(?!\\G)(?=\\\\\\1(?!#))|(?=\"\\1)",
					"name": "comment",
					"patterns": [ { "include": "#literals-string-raw-string-guts" } ]
				},
				{
					"begin": "(?=\\\\(#+)(?!#))",
					"end": "(?!\\G)(?=\\\\\\1(?!#))|(?=\"\\1)",
					"name": "comment",
					"patterns": [ { "include": "#literals-string-raw-string-guts" } ]
				}
			]
		},
		"literals-string-raw-string-guts": {
			"comment": "the same as #string-guts but with # in escapes",
			"patterns": [
				{
					"name": "constant.character.escape.swift",
					"match": "\\G\\\\#+[0\\\\tnr\"']"
				},
				{
					"name": "constant.character.escape.unicode.swift",
					"match": "\\G\\\\#+u\\{[0-9a-fA-F]{1,8}\\}"
				},
				{
					"contentName": "source.swift",
					"name": "meta.embedded.line.swift",
					"begin": "\\G\\\\#+\\(",
					"end": "(\\))",
					"beginCaptures": { "0": { "name": "punctuation.section.embedded.begin.swift" } },
					"endCaptures": {
						"0": { "name": "punctuation.section.embedded.end.swift" },
						"1": { "name": "source.swift" }
					},
					"patterns": [
						{
							"comment": " --> $self <-- ",
							"include": "source.swift"
						},
						{
							"comment": "Nested parens",
							"begin": "\\(",
							"end": "\\)"
						}
					]
				},
				{
					"name": "invalid.illegal.escape-not-recognized",
					"match": "\\G\\\\#+."
				}
			]
		}
	}
}

a

b##"\#\###\##t\###\##z\#

c
\##t
d
\##z
e
\##(
func foo<T>(x: T) async throws -> Int where T: Equatable {}
)
f
\#
g
\##u{0123}
h
\###
i


j"##

k

[jtbandes]

Wow that's quite cursed, congrats 🫠 It's gonna take me a while to wrap my head around this. It's also been 4 years since I added raw string support so I've probably forgotten some context.

Sounds like you've answered your question yourself but I believe the limitations I was referring to was that I didn't think it was possible to reference groups from begin inside the scope content pattern itself (so that begin=####" would require escapes to be like \####(...) inside the scope)

Does your solution handle things like ##" \##(this is interpolated) \#(but this is not interpolated) "## ?

[RedCMD]

Does your solution handle things like...

\#( & \###( will be treated as raw string
\##( will be recognized as the source.swift

I'm assuming that delimiters are not recognized inside \##( ... )?

also
should be able to remove the exception from the first pattern
as long as that include can be removed

"begin": "(?<=(?>[^#]|^)(#+)\")\\G",
"end": "(?=\\\\\\1(?!#))|(?=\"\\1)",
"name": "comment"

[RedCMD]

better exception handler

"swift-raw-strings": {
	"begin": "(#+)\"",
	"end": "\"\\1(#*)",
	"beginCaptures": { "0": { "name": "punctuation.definition.string.begin.raw.swift" } },
	"endCaptures": {
		"0": { "name": "punctuation.definition.string.end.raw.swift" },
		"1": { "name": "invalid.illegal.extra-closing-delimiter.swift" }
	},
	"name": "string.quoted.double.single-line.raw.swift",
	"patterns": [
		{
			"begin": "(?<=(?>[^#]|^)(#+)\")\\G(?!\\\\\\1(?!#))",
			"end": "(?=\\\\\\1(?!#))|(?=\"\\1)",
			"patterns": [
				{
					"name": "invalid.illegal.returns-not-allowed.swift",
					"match": "\\r|\\n"
				}
			]
		},
		{
			"begin": "(?=\\\\(#+)(?!#))",
			"end": "(?!\\G)(?=\\\\\\1(?!#))|(?=\"\\1)",
			"patterns": [
				{
					"name": "invalid.illegal.returns-not-allowed.swift",
					"match": "\\r|\\n"
				},
				{ "include": "#literals-string-raw-string-guts" }
			]
		}
	]
}