Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support TypeScript #51

Closed
Thom1729 opened this issue Dec 2, 2018 · 79 comments
Closed

Support TypeScript #51

Thom1729 opened this issue Dec 2, 2018 · 79 comments

Comments

@Thom1729
Copy link
Owner

Thom1729 commented Dec 2, 2018

Like it says. I'm not sure how much of an effort this would be.

Known bugs/missing features

  • Old-style casting (<Foo> bar). This conflicts with JSX, so I'll have to make sure that this is configurable. Implemented in beta.3
  • Type arguments in method calls. On the surface, this is ambiguous with ordinary JS syntax. I'll have to look into exactly how TypeScript parses this. May require branching.
  • Type arguments in template strings. Same concerns as above. May also require a special case for Styled Components.
  • Better detection of arrow function argument lists. May require branching.
  • Arrow function type parameters. Almost certainly require branching.
  • Directives in comments. For this, I'll need to figure out a general system for extending comments that works for other extensions as well.
@TehShrike
Copy link

It looks like MS maintains a TextMate language definition for TS: https://github.com/microsoft/TypeScript-TmLanguage/blob/master/TypeScript.YAML-tmLanguage

How much modification did you have to do to the original JavaScript language definition you pulled in?

@Thom1729
Copy link
Owner Author

I'm not 100% sure I understand the question, so I'll try to over-answer.

JS Custom is based directly on Sublime's core JavaScript syntax. To produce customized syntax definitions, JS Custom starts with the core syntax and applies the user-selected syntax extensions. Other than what is added or modified by those extensions, the resulting syntax is identical to the core JavaScript syntax. So in a sense, I have not modified that syntax at all. This was one of the key design considerations when designing JS Custom — I didn't want it to diverge from core over time, so it's set up so that I can easily “rebase” by bundling a newer core syntax with the package.

Microsoft's TypeScript syntax definition, on the other hand, is totally separate and incompatible with the core Sublime JavaScript definition. if they ever shared a historical ancestor, then they diverged a long time ago. Microsoft's implementation is an old-style TextMate syntax. It's unsophisticated and not particularly reliable, and it relies on inefficient regexp lookbehinds. Sublime's core JavaScript definition, on the other hand, is effectively a full parser; it's designed to handle any valid JavaScript code, with only a handful of known exceptions. It is also faster.

Adding TypeScript to JS Custom support would mean writing a new extension to handle TypeScript-specific features. In increasing order of effort, this means:

  1. Actually writing the extension.
  2. Writing documentation and a comprehensive test suite.
  3. Deciphering Microsoft's language spec.

The major hassle is that, while TypeScript is obviously an extension of JavaScript, Microsoft treats it as an entirely separate language. As far as I can tell, there is no official reference describing the new TypeScript syntax in terms of the original JavaScript syntax, so I'll have to reverse-engineer it from the full grammar in the TypeScript spec. This is both tedious and vexing.

If anyone has a good, authoritative-ish reference describing the difference between JavaScript and TypeScript, that would move the expected timeline of this feature from “someday” to “the next time I have a free weekend”.

@TehShrike
Copy link

Thanks for the excellent answer!

JS Custom is based directly on Sublime's core JavaScript syntax

That sounds good – I saw the "Derived from JavaScript Next" comment and thought that maybe you'd had to do some transformations to the original.

I was hoping that I might be able to just fork this module and drop in the TypeScript version of the TM language definition, but your answer makes me think that this module's code might be pretty coupled to the way that the TM definition is laid out? I haven't read this library's source yet.

The only feature I need is custom_templates.tags – I might try reading the source and see if I can start to noodle out what it would take to get a TypeScript fork working with just that feature.

Before I dive in, can you give me any guidance as to how the library code is coupled to the language definition?

@Thom1729
Copy link
Owner Author

The base syntax definition in JS Custom is an exact copy of Sublime's core JavaScript syntax.

Years ago, the third-party JavaScript Next package offered much better highlighting than the core JavaScript package, so at some point the core package adopted JavaScript Next as a starting point and moved forward from there. By now, I doubt that more than a handful of lines of code remain from JavaScript Next, but the comment remains acknowledging the original contribution.

N.B. The core JavaScript syntax is not a TextMate-compatible definition. It uses the newer, more powerful sublime-syntax format.

I was hoping that I might be able to just fork this module and drop in the TypeScript version of the TM language definition, but your answer makes me think that this module's code might be pretty coupled to the way that the TM definition is laid out? I haven't read this library's source yet.

Yes. Here's an example of how JS Custom works. The core syntax begins as follows:

%YAML 1.2
---
# Derived from JavaScript Next: https://github.com/Benvie/JavaScriptNext.tmLanguage
name: JavaScript
<more stuff>

If a JS Custom configuration uses the hidden option, then when generating the syntax JS Custom will use the hidden extension, which is defined as follows:

%YAML 1.2
%TAG ! tag:yaml-macros:yamlmacros.lib.extend:
---
!merge
hidden: true

The implementation details aren't really important; what matters is that the resulting syntax will begin like this:

%YAML 1.2
---
# Derived from JavaScript Next: https://github.com/Benvie/JavaScriptNext.tmLanguage
name: JavaScript
<more stuff>
hidden: true

The hidden extension is very simple and doesn't depend on the details of the core syntax. But consider the es_pipeline extension:

%YAML 1.2
%TAG ! tag:yaml-macros:yamlmacros.lib.extend:
---
!merge
contexts: !merge
  binary-operators: !prepend
    - match: '\|>'
      scope: keyword.operator.pipeline.js
      push: expression-begin

This modifies the core syntax's binary-operators context, and it uses the expression-begin context. It is extremely tightly coupled to the core syntax's implementation. And es_pipeline is one of the simplest extensions; extensions like custom_templates, 'flow', and jsx are a lot more complicated and are tied into the core syntax in many, many places.

This is one reason why JS Custom bundles the core syntax in the package rather than referring to the original copy in the JavaScript package — a given version of JS Custom is tied to an exact version of the core syntax, any changes to the base syntax could break JS Custom, and in order to update JS Custom to a newer core syntax I have to run the JS Custom tests and possibly fix bugs. This is not as inconvenient as it sounds, because I'm also the primary maintainer of the core JavaScript syntax.

If you had a JS Custom-like system that used Microsoft's TypeScript syntax as a base, you would have to write all of the extensions completely from scratch. In addition, because the Microsoft syntax uses the old TextMate system, it may not be possible to port every feature you want. By comparison, adding TypeScript support to JS Custom would be tremendously easier and a lot less work. It would also produce higher-quality highlighting, because Sublime's core JavaScript syntax is of much higher quality than Microsoft's TypeScript syntax.

On the other hand, if you like Microsoft's TypeScript syntax, and you don't mind its bugs and quirks, and all you want is highlighting of a few custom templates, then you could hand-edit that syntax definition to add the specific features you want. I'm not sure how much the old TextMate system would be a limiting factor — I haven't touched that format in years — but it's probably possible to get a couple of custom templates working reasonably well.

@tsujp
Copy link

tsujp commented Apr 21, 2020

Any update on this?

@Thom1729
Copy link
Owner Author

v2.4.0-alpha.2 is out with basic TypeScript support. It is probably buggy and missing important features, but it should be usable and — perhaps more importantly — it shouldn't break non-Typescript highlighting.

Please give it a try and report any issues in this thread. I expect to have time to finish this up over the next couple weeks, and any feedback would be invaluable.

@predragnikolic
Copy link

predragnikolic commented May 16, 2020

I cloned the repo and checkout the typescript branch.

When sublime started I run JSCustom: Rebuild Syntaxes,
and I expected to see Set Syntax: JS Custom - typescript in the command palette,
but I see Set Syntax: JS Custom - react and Set Syntax: JS Custom - default

How do i set the typescript syntax? :)

this is the output of the rebuild syn taxes command (if it is relevant):

Building JS Custom.sublime-syntax.yaml-macros... (/home/predrag/.config/sublime-text/Packages/JSCustom/src/syntax/JS Custom.sublime-syntax.yaml-macros)
Compiled to React.sublime-syntax.temp-34kizp. (/home/predrag/.config/sublime-text/Packages/User/JS Custom/Syntaxes/React.sublime-syntax.temp-34kizp)
[Succeeded in 3.81 seconds.]

Building JS Custom.sublime-syntax.yaml-macros... (/home/predrag/.config/sublime-text/Packages/JSCustom/src/syntax/JS Custom.sublime-syntax.yaml-macros)
Compiled to ~embed.sublime-syntax.temp-927x39. (/home/predrag/.config/sublime-text/Packages/User/JS Custom/Syntaxes/~embed.sublime-syntax.temp-927x39)
[Succeeded in 7.68 seconds.]

Building JS Custom.sublime-syntax.yaml-macros... (/home/predrag/.config/sublime-text/Packages/JSCustom/src/syntax/JS Custom.sublime-syntax.yaml-macros)
Compiled to Default.sublime-syntax.temp-_h1fml. (/home/predrag/.config/sublime-text/Packages/User/JS Custom/Syntaxes/Default.sublime-syntax.temp-_h1fml)
[Succeeded in 3.94 seconds.]

this ~embed.sublime-syntax.temp looks suspicious(but I am probably wrong) :)

@Thom1729
Copy link
Owner Author

I haven't added a default configuration with TypeScript enabled. I'll probably add one into an alpha or beta build, maybe when I work on the documentation. For now, you can create a custom configuration with the typescript option set to true:

"TypeScript": {
            "file_extensions": [ "ts", "tsx" ],
            "typescript": true,
        }

The ~embed configuration is a sort of hack. It's possible to use custom_templates to embed JavaScript highlighting within template strings. However, this can lead to infinite recursion and strange errors in certain circumstances. The ~embed configuration exists so that other first- and third-party syntaxes that embed the source.js scope (like HTML or Markdown) won't run into this issue.

The temp- suffixes are there because when JS Custom rebuilds an existing configuration, it replaces the old compiled syntax atomically by writing the new syntax to a temporary file, then replacing the old file with the new one. This prevents a Sublime-generated error popup if something goes wrong with the build.

@predragnikolic
Copy link

Thanks for the clarification :)

@Thom1729
Copy link
Owner Author

@predragnikolic Have you had a chance to test out the TypeScript support? I'm not using TypeScript at work right now, so I'm sure there are bugs that I won't run into myself.

@predragnikolic
Copy link

I am using typescript for only one personal project, that said I haven't used it that much. :)
I just noticed only thing that can be improved with conditional types.

type TypeName<T> =
    T extends string ? "string" :
//    ^^^^^^^ is source.js.typescript variable.other.readwrite.js ... I thing it should be a different scope 
    T extends number ? "number" :
    T extends boolean ? "boolean" :
    T extends undefined ? "undefined" :
    T extends Function ? "function" :
    "object";


function foo<T extends string | undefined>(baz: T): T {
//             ^^^^^^^ source.js.typescript meta.function.declaration.js meta.generic.js storage.modifier.extends.js
    return bar
}

the extends keyword should be highlighted in conditional types, but currently it is not:

2020-05-21-220513_556x412_scrot

If you don't mind, I can write on the ST discord channel, I am pretty sure that there are people(who use ts daily) and would like to try out the new ts support?
Personally, I found the JSCustom package really great and find it even more awesome that it wants to support typescript as well.

@Thom1729
Copy link
Owner Author

v2.4.0-alpha.5 should support conditional types.

Feel free to spread the word if you like. I think it's probably usable enough that more eyes on it would help.

@alecmev
Copy link

alecmev commented May 21, 2020

v2.4.0-alpha.2 is out with basic TypeScript support

Thank you! I've given it a quick go with a couple of type-heavy files, comparing with VSCode, and it's quite solid! The only thing that stands out is multi-line arrow functions, which I have quite a few of.

image

I know that they are tricky to scope and it isn't possible to detect their parameters properly right now (right?), but it can probably be assumed that something coming after an : in a scope like in the screenshot above is likely a type? Or is this not a safe assumption?

And shouldn't b in the first line be scoped as variable.parameter.function.js?

@Thom1729
Copy link
Owner Author

Arrow functions argument lists are tricky. The Sublime prereleases have a new branching feature that make them much easier and reliable to handle, but JS Custom is currently using the stable syntax for compatibility. Ideally, I'd like to provide TypeScript support for the stable release, but I haven't decided exactly how to handle arrow function argument lists.

There will definitely be a future-only version that uses branching to provide accurate arrow function parsing (including type annotations).

@alecmev
Copy link

alecmev commented May 21, 2020

Looking forward to ST4!

A bit off-topic, but relevant: Is it okay to have both typescript and flow enabled in one config? Any downsides to look out for? Performance? Similarly, would I want to avoid using a TS-enabled config for highlighting JS-only files?

@Thom1729
Copy link
Owner Author

TypeScript and Flow would conflict; when it saw a type annotation, it would use one or the other. I wouldn't be surprised if the resulting behavior were glitchy.

Otherwise, there should be no harm in enabling Flow or TypeScript for vanilla-JS files. Performance shouldn't be an issue; Sublime's syntax engine is extremely fast.

@Thom1729
Copy link
Owner Author

v2.4.0-alpha.6 is out with support for namespaces, better object types, and various fixes and improvements.

Incidentally, I have just learned that the language specification linked from the TypeScript project repository, which I've been using as a reference, hasn't been updated in four years. No newer version exists, the TypeScript maintainers do not consider it a priority, and there are no plans to update it (microsoft/TypeScript#15711).

I'm still hoping to get TypeScript support up to beta-quality within the next couple of weeks, but in the absence of a language spec, I expect to be chasing down bugs for months. I'll most likely have to release beta-quality TS support in stable builds for a while with appropriate warnings.

@alecmev
Copy link

alecmev commented May 22, 2020

Not sure what your workflow is like, but maybe Babel’s tests could be of help somehow? Or some other part of the project.

@Thom1729
Copy link
Owner Author

v2.4.0-alpha.7 is out, implementing various features from TypeScript versions 1.9 through 2.8.

@Thom1729
Copy link
Owner Author

FYI, currently my workflow is going through the release notes in order (because some features are otherwise undocumented) with AST Explorer open in another tab (because the only authoritative reference for syntax is the parser itself).

@Thom1729
Copy link
Owner Author

v2.4.0-alpha.8 is out, which should be current up to TypeScript 3.9. Known missing features include:

  • Type arguments to functions, template strings, etc.
  • Comment directives.
  • Builtin utility types.
  • Old-style type assertions.

@tsujp
Copy link

tsujp commented May 27, 2020

Am I doing something incorrect here? I've cloned and checked out to tag v2.4.0-alpha.8 within $HOME/.config/sublime-text-3/Package and I cannot use JSCustom.

@tsujp
Copy link

tsujp commented May 27, 2020

I got it working now, as in installed, but it seems to not be playing nicely with TypeScript and React, here are my settings:

{
  "defaults": {
    "custom_template_tags": false,
    "flow_types": false,
    "jsx": false,
  },

  "configurations": {
    "Default": {},
    "React": {
      "file_extensions": [ "js", "jsx", "tsx", "ts" ],
      "typescript": true,
      "jsx": true,
    }
  },

  "embed_configuration": {
    "name": "JS Custom (Embedded)",
    "scope": "source.tsx",
    "hidden": true,
    "file_extensions": [],
    "custom_template_tags": true,
    "custom_templates": true,
    "styled_components": true,
  },

  "auto_build": true,
  "jsx_close_tag": true
}

Screenshot:

ss

@Thom1729
Copy link
Owner Author

It doesn't seem to be using TypeScript at all. Try rebuilding your syntaxes.

@tsujp
Copy link

tsujp commented May 28, 2020

I've rebuilt syntaxes and reinstalled JSCustom to no avail, this is in ST4 too.

@tsujp
Copy link

tsujp commented May 28, 2020

This is the output TypeScript.sublime-syntax it generates:

Click for file contents
%YAML 1.2
---
scope: source.js.typescript
variables:
  bin_digit: '[01_]'
  oct_digit: '[0-7_]'
  dec_digit: '[0-9_]'
  hex_digit: '[\h_]'
  dec_integer: (?:0|[1-9]{{dec_digit}}*)
  dec_exponent: (?:[Ee](?:[-+]|(?![-+])){{dec_digit}}*)

  identifier_escape: (?:\\u(?:\h{4}|\{\h+\}))
  identifier_start: (?:[_$\p{L}\p{Nl}]|{{identifier_escape}})
  identifier_part: (?:[_$\p{L}\p{Nl}\p{Mn}\p{Mc}\p{Nd}\p{Pc}\x{200C}\x{200D}]|{{identifier_escape}})
  identifier_break: (?!{{identifier_part}})

  identifier: (?:{{identifier_start}}{{identifier_part}}*{{identifier_break}})
  constant_identifier: (?:[[:upper:]]{{identifier_part}}*{{identifier_break}})
  dollar_only_identifier: (?:\${{identifier_break}})
  dollar_identifier: (?:(\$){{identifier_part}}*{{identifier_break}})

  block_comment_contents: (?:(?:[^*]|\*(?!/))*)
  block_comment: (?:/\*{{block_comment_contents}}\*/)
  nothing: (?x:(?:\s+|{{block_comment}})*)
  line_ending_ahead: (?={{nothing}}(?:/\*{{block_comment_contents}})?$)

  # https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Lexical_grammar
  reserved_word: |-
    (?x:
      break|case|catch|class|const|continue|debugger|default|delete|do|else|
      export|extends|finally|for|function|if|import|in|instanceof|new|return|
      super|switch|this|throw|try|typeof|var|void|while|with|yield|
      enum|
      null|true|false
    ){{identifier_break}}

  non_reserved_identifier: (?:(?!{{reserved_word}}){{identifier}})

  either_func_lookahead: (?:{{func_lookahead}}|{{arrow_func_lookahead}})
  binding_pattern_lookahead: (?:{{identifier}}|\[|\{)
  left_expression_end_lookahead: (?!\s*[.\[\(])

  property_name: >-
    (?x:
      {{identifier}}
      | '(?:[^\\']|\\.)*'
      | "(?:[^\\"]|\\.)*"
      | \[ .* \]
    )

  class_element_name: |-
    (?x:
      \*?
      {{property_name}}
      | \#{{identifier}}
    )

  func_lookahead: |-
    (?x:
      \s*
      (?:async{{identifier_break}}{{nothing}})?
      function{{identifier_break}}
    )

  arrow_func_lookahead: |-
    (?x:
      \s*
      (?:async\s*)?
      (?:
        {{identifier}}
        | \( ( [^()] | \( [^()]* \) )* \)
      )
      \s*
      =>
    )

  method_lookahead: |-
    (?x:(?=
      (?: get|set|async ){{identifier_break}}(?!\s*:)
      | \*
      | {{property_name}} \s* \(
    ))

  line_continuation_lookahead: >-
    (?x:(?=
      \s*
      (?! \+\+ | -- )
      (?=
        != |
        [-+*/%><=&|^\[(;,.:?] |
        (?:in|instanceof){{identifier_break}}
      )
    ))

  dot_accessor: |-
    (?x: # Match . and .?, but not .?( or .?[
      \.
      (?! \? [\[(] )
      \??
    )

file_extensions:
  - ts
  - tsx
first_line_match: ^#!\s*/.*\b(node|js)\b
contexts:
  main:
    - include: comments-top-level

    - match: \)|\}|\]
      scope: invalid.illegal.stray-bracket-end.js
      # Don't pop or embedding could break.

    - include: statements

  prototype:
    - include: comments

  comments:
    - match: /\*\*(?!/)
      scope: punctuation.definition.comment.begin.js
      push:
        - meta_include_prototype: false
        - meta_scope: comment.block.documentation.js
        - match: \*/
          scope: punctuation.definition.comment.end.js
          pop: true
        - match: ^\s*(\*)(?!/)
          captures:
            1: punctuation.definition.comment.js
    - match: /\*
      scope: punctuation.definition.comment.begin.js
      push:
        - meta_include_prototype: false
        - meta_scope: comment.block.js
        - match: \*/
          scope: punctuation.definition.comment.end.js
          pop: true
    - match: //
      scope: punctuation.definition.comment.js
      push:
        - meta_include_prototype: false
        - meta_scope: comment.line.double-slash.js
        - match: \n
          pop: true

  comments-top-level:
    - match: ^(#!).*$\n?
      scope: comment.line.shebang.js
      captures:
        1: punctuation.definition.comment.js

  else-pop:
    - match: (?=\S)
      pop: true

  immediately-pop:
    - match: ''
      pop: true

  comma-separator:
    - match: ','
      scope: punctuation.separator.comma.js

  import-export:
    - match: import{{identifier_break}}(?!{{nothing}}[.(])
      scope: keyword.control.import-export.js
      set:
        - meta_scope: meta.import.js
        - match: (?=[.(]) # Recovery for import expressions
          set:
            - expression-statement-end
            - import-expression-end
        - match: (?=\S)
          set:
            - import-meta
            - expect-semicolon
            - import-string-or-items

    - match: export{{identifier_break}}
      scope: keyword.control.import-export.js
      set:
        - export-meta
        - export-extended

  import-meta:
    - meta_include_prototype: false
    - meta_scope: meta.import.js
    - include: immediately-pop

  import-export-alias:
    - match: as{{identifier_break}}
      scope: keyword.control.import-export.js
      set:
        - match: default{{identifier_break}}
          scope: keyword.control.import-export.js
          pop: true
        - match: '{{identifier}}'
          scope: variable.other.readwrite.js
          pop: true
        - include: else-pop
    - include: else-pop

  import-export-from:
    - match: from{{identifier_break}}
      scope: keyword.control.import-export.js
      set: literal-string
    - include: else-pop

  import-string-or-items:
    - include: literal-string
    - match: (?=\S)
      set:
        - import-export-from
        - import-list
        - import-export-alias
        - import-item

  import-list:
    - match: ','
      scope: punctuation.separator.comma.js
      push:
        - import-export-alias
        - import-item
    - include: else-pop

  import-item:
    - match: \{
      scope: punctuation.section.block.begin.js
      set: import-brace
    - match: '{{non_reserved_identifier}}'
      scope: variable.other.readwrite.js
      pop: true
    - match: \*
      scope: constant.other.js
      pop: true
    - include: else-pop

  import-brace:
    - meta_scope: meta.block.js
    - include: comma-separator
    - match: \}
      scope: punctuation.section.block.end.js
      pop: true
    - match: '{{identifier}}'
      scope: variable.other.readwrite.js
      push: import-export-alias
    - match: \*
      scope: constant.other.js
      push: import-export-alias
    - include: else-pop

  export-meta:
    - meta_include_prototype: false
    - meta_scope: meta.export.js
    - include: immediately-pop

  export-extended:
    - include: variable-declaration
    - include: function-or-class-declaration

    - match: default{{identifier_break}}
      scope: keyword.control.import-export.js
      set:
        - include: function-or-class-declaration
        - match: (?=\S)
          set: expression-statement

    - match: (?=\S)
      set:
        - expect-semicolon
        - import-export-from
        - export-list
        - import-export-alias
        - export-item

  export-list:
    - match: ','
      scope: punctuation.separator.comma.js
      push:
        - import-export-alias
        - export-item
    - include: else-pop

  export-item:
    - match: \{
      scope: punctuation.section.block.begin.js
      set: export-brace
    - match: '{{non_reserved_identifier}}'
      scope: variable.other.readwrite.js
      pop: true
    - match: \*
      scope: constant.other.js
      pop: true
    - include: else-pop

  export-brace:
    - meta_scope: meta.block.js
    - include: comma-separator
    - match: \}
      scope: punctuation.section.block.end.js
      pop: true
    - match: '{{identifier}}'
      scope: variable.other.readwrite.js
      push: import-export-alias
    - match: \*
      scope: constant.other.js
      push: import-export-alias
    - include: else-pop

  statements:
    - match: \)|\}|\]
      scope: invalid.illegal.stray-bracket-end.js
      pop: true

    - match: (?=\S)
      push: statement

  statement:
    - match: \;
      scope: punctuation.terminator.statement.empty.js
      pop: true

    - include: import-export
    - include: conditional
    - include: block
    - include: label
    - include: variable-declaration

    - match: break{{identifier_break}}
      scope: keyword.control.flow.break.js
      set:
        - expect-semicolon
        - expect-label

    - match: continue{{identifier_break}}
      scope: keyword.control.flow.continue.js
      set:
        - expect-semicolon
        - expect-label

    - match: debugger{{identifier_break}}
      scope: keyword.control.flow.debugger.js
      set: expect-semicolon

    - match: return{{identifier_break}}
      scope: keyword.control.flow.return.js
      set: restricted-production

    - match: throw{{identifier_break}}
      scope: keyword.control.flow.throw.js
      set: restricted-production

    - include: function-or-class-declaration

    - include: decorator

    - include: expression-statement

  expect-semicolon:
    - match: \;
      scope: punctuation.terminator.statement.js
      pop: true
    - include: else-pop

  expect-label:
    - meta_include_prototype: false
    - match: (?={{nothing}}{{identifier}})
      set:
        - match: '{{non_reserved_identifier}}'
          scope: variable.label.js
          pop: true
        - match: '{{identifier}}'
          scope: invalid.illegal.identifier.js variable.label.js
          pop: true
        - include: else-pop
    - include: immediately-pop

  block:
    - match: \{
      scope: punctuation.section.block.begin.js
      set:
        - meta_scope: meta.block.js
        - match: \}
          scope: punctuation.section.block.end.js
          pop: true
        - include: statements

  variable-binding-pattern:
    - include: variable-binding-name
    - include: variable-binding-array-destructuring
    - include: variable-binding-object-destructuring
    - include: else-pop

  variable-binding-name:
    - match: (?={{non_reserved_identifier}})
      set:
        - meta_scope: meta.binding.name.js
        - include: literal-variable

  variable-binding-array-destructuring:
    - match: \[
      scope: punctuation.section.brackets.begin.js
      set:
        - meta_scope: meta.binding.destructuring.sequence.js
        - match: \]
          scope: punctuation.section.brackets.end.js
          pop: true
        - include: variable-binding-spread
        - include: variable-binding-list

  variable-binding-object-destructuring:
    - match: \{
      scope: punctuation.section.block.begin.js
      set:
        - meta_scope: meta.binding.destructuring.mapping.js
        - match: \}
          scope: punctuation.section.block.end.js
          pop: true
        - include: variable-binding-spread
        - match: (?={{identifier}}|\[|'|")
          push:
            - initializer
            - variable-binding-object-alias
            - object-literal-meta-key
            - variable-binding-object-key
        - include: comma-separator

  variable-binding-object-alias:
    - match: ':'
      scope: punctuation.separator.key-value.js
      set: variable-binding-pattern
    - include: else-pop

  variable-binding-object-key:
    - match: '{{identifier}}(?=\s*:)'
      pop: true
    - include: literal-string
    - include: computed-property-name
    - include: variable-binding-name
    - include: else-pop

  variable-binding-spread:
    - match: \.\.\.
      scope: keyword.operator.spread.js
      push: variable-binding-pattern

  variable-binding-list:
    - include: comma-separator
    - match: (?={{binding_pattern_lookahead}})
      push:
        - initializer
        - variable-binding-pattern
    - include: else-pop

  variable-binding-top:
    - include: function-assignment
    - match: (?={{binding_pattern_lookahead}})
      set:
        - initializer
        - variable-binding-pattern
    - include: else-pop

  variable-binding-list-top:
    - match: '{{line_ending_ahead}}'
      set:
        - match: '{{line_continuation_lookahead}}'
          set: variable-binding-top
        - include: else-pop
    - match: ','
      scope: punctuation.separator.comma.js
      push: variable-binding-top
    - include: else-pop

  variable-declaration:
    - match: (?:const|let|var){{identifier_break}}
      scope: storage.type.js
      set:
        - expect-semicolon
        - variable-binding-list-top
        - variable-binding-top

  function-parameter-binding-pattern:
    - include: function-parameter-binding-name
    - include: function-parameter-binding-array-destructuring
    - include: function-parameter-binding-object-destructuring
    - include: else-pop

  function-parameter-binding-name:
    - match: '{{non_reserved_identifier}}'
      scope: meta.binding.name.js variable.parameter.function.js
    - match: '{{identifier}}'
      scope: invalid.illegal.identifier.js meta.binding.name.js variable.parameter.function.js

  function-parameter-binding-array-destructuring:
    - match: \[
      scope: punctuation.section.brackets.begin.js
      set:
        - meta_scope: meta.binding.destructuring.sequence.js
        - match: \]
          scope: punctuation.section.brackets.end.js
          pop: true
        - include: function-parameter-binding-list

  function-parameter-binding-object-destructuring:
    - match: \{
      scope: punctuation.section.block.begin.js
      set:
        - meta_scope: meta.binding.destructuring.mapping.js
        - match: ','
          scope: punctuation.separator.parameter.function.js
        - match: \}
          scope: punctuation.section.block.end.js
          pop: true
        - include: function-parameter-binding-spread
        - match: (?={{identifier}}|\[|'|")
          push:
            - initializer
            - function-parameter-binding-object-alias
            - object-literal-meta-key
            - function-parameter-binding-object-key

  function-parameter-binding-object-alias:
    - match: ':'
      scope: punctuation.separator.key-value.js
      set: function-parameter-binding-pattern
    - include: else-pop

  function-parameter-binding-object-key:
    - match: '{{identifier}}(?=\s*:)'
      pop: true
    - include: literal-string
    - include: computed-property-name
    - include: function-parameter-binding-name
    - include: else-pop

  function-parameter-binding-spread:
    - match: \.\.\.
      scope: keyword.operator.spread.js
      push: function-parameter-binding-pattern

  function-parameter-binding-list:
    - match: ','
      scope: punctuation.separator.parameter.function.js
    - include: function-parameter-binding-spread
    - match: (?={{binding_pattern_lookahead}})
      push:
        - initializer
        - function-parameter-binding-pattern
    - include: else-pop

  function-or-class-declaration:
    - match: (?=class{{identifier_break}})
      set: class

    - match: (?={{func_lookahead}})
      set: function-declaration

  initializer:
    - match: '='
      scope: keyword.operator.assignment.js
      set: expression-no-comma
    - include: else-pop

  function-initializer:
    - meta_scope: meta.function.declaration.js
    - match: '='
      scope: keyword.operator.assignment.js
      set:
        - meta_content_scope: meta.function.declaration.js
        - include: expression-no-comma

    - include: else-pop

  expression-statement:
    - match: (?=\S)
      set:
        - expect-semicolon
        - expression-statement-end
        - expression-begin

  expression-statement-end:
    - match: '{{line_ending_ahead}}'
      set:
        - match: '{{line_continuation_lookahead}}'
          set: expression-statement-end
        - include: else-pop
    - include: expression-end

  restricted-production:
    - meta_include_prototype: false
    - match: '{{line_ending_ahead}}'
      pop: true
    - match: ''
      set: expression-statement

  expect-case-colon:
    - match: ':'
      scope: punctuation.separator.js
      pop: true
    - include: else-pop

  conditional:
    - match: switch{{identifier_break}}
      scope: keyword.control.conditional.switch.js
      set:
        - switch-meta
        - switch-block
        - expect-parenthesized-expression

    - match: do{{identifier_break}}
      scope: keyword.control.loop.do-while.js
      set:
        - do-while-meta
        - do-while-condition
        - block-scope

    - match: for{{identifier_break}}
      scope: keyword.control.loop.for.js
      set:
        - for-meta
        - block-scope
        - for-condition
        - for-await

    - match: while{{identifier_break}}
      scope: keyword.control.loop.while.js
      set:
        - while-meta
        - block-scope
        - expect-parenthesized-expression

    - match: with{{identifier_break}}
      scope: keyword.control.import.with.js
      set:
        - with-meta
        - block-scope
        - expect-parenthesized-expression

    - match: if{{identifier_break}}
      scope: keyword.control.conditional.if.js
      set:
        - conditional-meta
        - block-scope
        - expect-parenthesized-expression

    - match: else\s+if{{identifier_break}}
      scope: keyword.control.conditional.elseif.js
      set:
        - conditional-meta
        - block-scope
        - expect-parenthesized-expression

    - match: else{{identifier_break}}
      scope: keyword.control.conditional.else.js
      set:
        - conditional-meta
        - block-scope

    - match: try{{identifier_break}}
      scope: keyword.control.exception.try.js
      set:
        - try-meta
        - block-scope

    - match: finally{{identifier_break}}
      scope: keyword.control.exception.finally.js
      set:
        - finally-meta
        - block-scope

    - match: catch{{identifier_break}}
      scope: keyword.control.exception.catch.js
      set:
        - catch-meta
        - block-scope
        - expect-parenthesized-expression

  expect-parenthesized-expression:
    - include: parenthesized-expression
    - include: else-pop

  switch-meta:
    - meta_include_prototype: false
    - meta_scope: meta.switch.js
    - include: immediately-pop

  do-while-meta:
    - meta_include_prototype: false
    - meta_scope: meta.do-while.js
    - include: immediately-pop

  for-meta:
    - meta_include_prototype: false
    - meta_scope: meta.for.js
    - include: immediately-pop

  while-meta:
    - meta_include_prototype: false
    - meta_scope: meta.while.js
    - include: immediately-pop

  with-meta:
    - meta_include_prototype: false
    - meta_scope: meta.with.js
    - include: immediately-pop

  conditional-meta:
    - meta_include_prototype: false
    - meta_scope: meta.conditional.js
    - include: immediately-pop

  try-meta:
    - meta_include_prototype: false
    - meta_scope: meta.try.js
    - include: immediately-pop

  finally-meta:
    - meta_include_prototype: false
    - meta_scope: meta.finally.js
    - include: immediately-pop

  catch-meta:
    - meta_include_prototype: false
    - meta_scope: meta.catch.js
    - include: immediately-pop

  for-await:
    - match: await{{identifier_break}}
      scope: keyword.control.flow.await.js
      pop: true
    - include: else-pop

  for-condition:
    - match: \(
      scope: punctuation.section.group.js
      set:
        - for-condition-end
        - for-condition-contents
    - include: else-pop

  for-condition-end:
    - meta_scope: meta.group.js

    - match: \)
      scope: punctuation.section.group.js
      pop: true

  for-condition-contents:
    # This could be either type of for loop.
    - match: (?:const|let|var){{identifier_break}}
      scope: storage.type.js
      set:
        -   - include: for-of-rest
            - match: (?=\S)
              set:
                - for-oldstyle-rest
                - variable-binding-list
                - initializer
        - variable-binding-pattern

    - match: (?=\S)
      set:
        -   - include: for-of-rest
            - match: (?=\S)
              set: for-oldstyle-rest
        - expression-end-no-in
        - expression-begin

  for-of-rest:
    - match: (?:of|in){{identifier_break}}
      scope: keyword.operator.word.js
      set: expression

  for-oldstyle-rest:
    - match: (?=\))
      pop: true
    - match: ;
      scope: punctuation.separator.expression.js
    - match: (?=\S)
      push: expression

  block-scope:
    - include: block
    - include: else-pop

  block-meta:
    - meta_include_prototype: false
    - meta_scope: meta.block.js
    - include: immediately-pop

  switch-block:
    - match: \{
      scope: punctuation.section.block.begin.js
      set: switch-block-contents
    - include: else-pop

  switch-block-contents:
    - meta_scope: meta.block.js

    - match: \}
      scope: punctuation.section.block.end.js
      pop: true

    - match: case{{identifier_break}}
      scope: keyword.control.conditional.case.js
      push:
        - expect-case-colon
        - expression

    - match: default{{identifier_break}}
      scope: keyword.control.conditional.default.js
      push:
        - expect-case-colon

    - include: statements

  do-while-condition:
    - match: while{{identifier_break}}
      scope: keyword.control.loop.while.js
      set: parenthesized-expression
    - include: else-pop

  decorator:
    - match: '@'
      scope: punctuation.definition.annotation.js
      push:
        - decorator-meta
        - decorator-expression-end
        - decorator-expression-begin

  decorator-meta:
    - meta_include_prototype: false
    - meta_scope: meta.annotation.js
    - include: immediately-pop

  decorator-name:
    - match: '{{identifier}}{{left_expression_end_lookahead}}'
      scope: variable.annotation.js
      pop: true

  decorator-expression-end:
    - match: '{{dot_accessor}}'
      scope: punctuation.accessor.js
      push:
        - include: decorator-name
        - include: object-property

    - include: left-expression-end

  decorator-expression-begin:
    - include: decorator-name
    - include: expression-begin

  expression-break:
    - match: (?=[;})\]])
      pop: true

  expression:
    - meta_include_prototype: false
    - match: ''
      set: [expression-end, expression-begin]

  expression-no-comma:
    - meta_include_prototype: false
    - match: ''
      set: [expression-end-no-comma, expression-begin]

  expression-list:
    - include: expression-break
    - include: comma-separator
    - match: (?=\S)
      push: expression-no-comma

  left-expression-end:
    - include: expression-break

    - match: (?=`)
      push: literal-string-template

    - match: (?=(?:\.\?)?\()
      push: function-call-arguments

    - include: property-access

    - include: fallthrough

    - include: else-pop

  expression-end:
    - include: postfix-operators
    - include: binary-operators
    - include: ternary-operator

    - include: left-expression-end

  expression-end-no-comma:
    - match: (?=,)
      pop: true
    - include: expression-end

  expression-end-no-in:
    - match: (?=in{{identifier_break}})
      pop: true
    - include: expression-end

  expression-begin:
    - include: expression-break

    - include: yield-expression
    - include: await-expression

    - include: regexp-complete
    - include: literal-string
    - include: tagged-template
    - include: literal-string-template
    - include: constructor
    - include: literal-number
    - include: prefix-operators
    - include: import-meta-expression

    - include: class
    - include: constants
    - include: function-assignment
    - include: either-function-declaration
    - include: object-literal

    - include: parenthesized-expression
    - include: array-literal

    - include: literal-call
    - include: literal-variable

    - include: else-pop

  fallthrough:
    # If an arrow function has the ( and ) on different lines, we won't have matched
    - match: =>
      scope: storage.type.function.arrow.js
      push:
        - function-meta
        - arrow-function-expect-body

  literal-string:
    - match: "'"
      scope: punctuation.definition.string.begin.js
      set:
        - meta_include_prototype: false
        - meta_scope: meta.string.js string.quoted.single.js
        - match: \'
          scope: punctuation.definition.string.end.js
          pop: true
        - match: \n
          scope: invalid.illegal.newline.js
          pop: true
        - include: string-content
    - match: '"'
      scope: punctuation.definition.string.begin.js
      set:
        - meta_include_prototype: false
        - meta_scope: meta.string.js string.quoted.double.js
        - match: \"
          scope: punctuation.definition.string.end.js
          pop: true
        - match: \n
          scope: invalid.illegal.newline.js
          pop: true
        - include: string-content

  tagged-template:
    - match: '{{identifier}}(?=\s*`)'
      scope: variable.function.tagged-template.js
      pop: true

  literal-string-template:
    - match: '`'
      scope: punctuation.definition.string.begin.js
      set:
        - meta_include_prototype: false
        - meta_scope: meta.string.js string.quoted.other.js
        - match: '`'
          scope: punctuation.definition.string.end.js
          pop: true
        - match: \$\{
          scope: punctuation.section.interpolation.begin.js
          push:
            - clear_scopes: 1
            - meta_scope: meta.interpolation.js
            - meta_content_scope: source.js.embedded
            - match: \}
              scope: punctuation.section.interpolation.end.js
              pop: true
            - match: (?=\S)
              push: expression
        - include: string-content

  string-content:
    - match: \\\n
      scope: constant.character.escape.newline.js
    - match: \\(?:x\h\h|u\h\h\h\h|.)
      scope: constant.character.escape.js

  regexp-complete:
    - match: /
      scope: punctuation.definition.string.begin.js
      set: regexp

  regexp:
    - meta_include_prototype: false
    - meta_scope: meta.string.js string.regexp.js
    - match: /
      scope: punctuation.definition.string.end.js
      set:
        - meta_include_prototype: false
        - meta_content_scope: meta.string.js string.regexp.js
        - match: '[gimyus]'
          scope: keyword.other.js
        - match: '[A-Za-z0-9]'   # Ignore unknown flags for future-compatibility
        - include: immediately-pop
    - match: (?=.|\n)
      push:
        - meta_include_prototype: false
        - match: (?=/)
          pop: true
        - include: scope:source.regexp.js

  constructor:
    - match: new{{identifier_break}}
      scope: keyword.operator.word.new.js
      set:
        - match: (?=\s*\.)
          set: new-target
        - match: (?=\s*\S)
          set:
            - constructor-meta
            - constructor-body-expect-arguments
            - constructor-body-expect-class-end
            - constructor-body-expect-class-begin

  constructor-meta:
    - meta_include_prototype: false
    - meta_scope: meta.function-call.constructor.js
    - include: immediately-pop

  constructor-body-expect-arguments:
    - include: function-call-arguments
    - include: else-pop

  constructor-body-expect-class-end:
    - include: property-access
    - include: else-pop

  constructor-body-expect-class-begin:
    - match: (?={{identifier}}\s*\()
      set:
        - include: support
        - match: '{{dollar_only_identifier}}'
          scope: variable.type.dollar.only.js punctuation.dollar.js
          pop: true
        - match: '{{dollar_identifier}}'
          scope: variable.type.dollar.js
          captures:
            1: punctuation.dollar.js
          pop: true
        - match: '{{identifier}}'
          scope: variable.type.js
          pop: true
        - include: else-pop

    - include: expression-begin

  new-target:
    - match: \.
      scope: punctuation.accessor.dot.js
      set:
        - match: target{{identifier_break}}
          scope: variable.language.target.js
          pop: true
        - include: else-pop

    - include: else-pop

  prefix-operators:
    - match: '~'
      scope: keyword.operator.bitwise.js
    - match: '!(?!=)'
      scope: keyword.operator.logical.js
    - match: --
      scope: keyword.operator.arithmetic.js
    - match: \+\+
      scope: keyword.operator.arithmetic.js
    - match: \.\.\.
      scope: keyword.operator.spread.js
    - match: \+|\-
      scope: keyword.operator.arithmetic.js
    - match: (?:delete|typeof|void){{identifier_break}}
      scope: keyword.operator.js

  binary-operators:
    - match: instanceof{{identifier_break}}
      scope: keyword.operator.js
      push: expression-begin
    - match: in{{identifier_break}}
      scope: keyword.operator.js
      push: expression-begin
    - match: '&&|\|\||\?\?'
      scope: keyword.operator.logical.js
      push: expression-begin
    - match: =(?![=>])
      scope: keyword.operator.assignment.js
      push: expression-begin
    - match: |-
        (?x)
        %=   | # assignment      right-to-left   both
        &=   | # assignment      right-to-left   both
        \*=  | # assignment      right-to-left   both
        \+=  | # assignment      right-to-left   both
        -=   | # assignment      right-to-left   both
        /=   | # assignment      right-to-left   both
        \^=  | # assignment      right-to-left   both
        \|=  | # assignment      right-to-left   both
        <<=  | # assignment      right-to-left   both
        >>=  | # assignment      right-to-left   both
        >>>=   # assignment      right-to-left   both
      scope: keyword.operator.assignment.augmented.js
      push: expression-begin
    - match: |-
        (?x)
        <<   | # bitwise-shift   left-to-right   both
        >>>  | # bitwise-shift   left-to-right   both
        >>   | # bitwise-shift   left-to-right   both
        &    | # bitwise-and     left-to-right   both
        \^   | # bitwise-xor     left-to-right   both
        \|     # bitwise-or      left-to-right   both
      scope: keyword.operator.bitwise.js
      push: expression-begin
    - match: |-
        (?x)
        <=   | # relational      left-to-right   both
        >=   | # relational      left-to-right   both
        <    | # relational      left-to-right   both
        >      # relational      left-to-right   both
      scope: keyword.operator.relational.js
      push: expression-begin
    - match: |-
        (?x)
        ===  | # equality        left-to-right   both
        !==  | # equality        left-to-right   both
        ==   | # equality        left-to-right   both
        !=     # equality        left-to-right   both
      scope: keyword.operator.comparison.js
      push: expression-begin
    - match: |-
        (?x)
        /    | # division        left-to-right   both
        %    | # modulus         left-to-right   both
        \*   | # multiplication  left-to-right   both
        \+   | # addition        left-to-right   both
        -      # subtraction     left-to-right   both
      scope: keyword.operator.arithmetic.js
      push: expression-begin
    - match: ','
      scope: keyword.operator.comma.js # Comma operator, not punctuation.
      push: expression-begin

  ternary-operator:
    - match: \?
      scope: keyword.operator.ternary.js
      set:
        - ternary-operator-expect-colon
        - expression-no-comma

  ternary-operator-expect-colon:
    - match: ':'
      scope: keyword.operator.ternary.js
      set: expression-no-comma
    - include: else-pop

  postfix-operators:
    - match: --
      scope: keyword.operator.arithmetic.js
    - match: \+\+
      scope: keyword.operator.arithmetic.js

  yield-expression:
    - match: yield{{identifier_break}}
      scope: keyword.control.flow.yield.js
      set:
        - match: $
          pop: true
        - match: \*
          scope: keyword.generator.asterisk.js
          set: expression-begin
        - match: (?=\S)
          set: expression-begin

  await-expression:
    - match: await{{identifier_break}}
      scope: keyword.control.flow.await.js

  class:
    - match: class{{identifier_break}}
      scope: storage.type.class.js
      set:
        - class-meta
        - class-body
        - class-extends
        - class-name

  class-meta:
    - meta_include_prototype: false
    - meta_scope: meta.class.js
    - include: immediately-pop

  class-body:
    - match: \{
      scope: punctuation.section.block.begin.js
      set: class-body-contents

    - include: else-pop

  class-body-contents:
    - meta_scope: meta.block.js

    - match: \}
      scope: punctuation.section.block.end.js
      pop: true

    - match: \;
      scope: punctuation.terminator.statement.js

    - include: decorator

    - match: constructor{{identifier_break}}
      scope: entity.name.function.constructor.js
      push:
        - function-meta
        - function-declaration-expect-body
        - function-declaration-meta
        - function-declaration-expect-parameters

    - match: static{{identifier_break}}
      scope: storage.modifier.js
      push: class-field

    - match: (?={{class_element_name}})
      push: class-field

  class-extends:
    - match: extends{{identifier_break}}
      scope: storage.modifier.extends.js
      set:
        - inherited-class-expression-end
        - inherited-class-expression-begin
    - include: else-pop

  inherited-class-name:
    - match: '{{non_reserved_identifier}}{{left_expression_end_lookahead}}'
      scope: entity.other.inherited-class.js
      pop: true

  inherited-class-expression-end:
    - match: '{{dot_accessor}}'
      scope: punctuation.accessor.js
      push:
        - include: inherited-class-name
        - include: object-property

    - include: left-expression-end

  inherited-class-expression-begin:
    - include: inherited-class-name
    - include: expression-begin

  class-name:
    - match: '{{non_reserved_identifier}}'
      scope: entity.name.class.js
      pop: true
    - include: else-pop

  class-field:
    - match: '{{method_lookahead}}'
      set: method-declaration

    - match: |-
        (?x)(?=
          \#? {{identifier}}
          \s* = \s*
          {{either_func_lookahead}}
        )
      set:
        - function-initializer
        - function-name-meta
        - literal-variable-base

    - match: (?=#?{{property_name}})
      set:
        - field-initializer-or-method-declaration
        - field-name

    - include: else-pop

  class-field-rest:
    - match: ','
      scope: punctuation.separator.js
      push:
        - initializer
        - field-name
    - include: else-pop

  field-initializer-or-method-declaration:
    - match: (?=\()
      set:
        - function-meta
        - function-declaration-expect-body
        - function-declaration-meta
        - function-declaration-expect-parameters
    - match: (?=\S)
      set:
        - class-field-rest
        - initializer

  constants:
    - match: true{{identifier_break}}
      scope: constant.language.boolean.true.js
      pop: true
    - match: false{{identifier_break}}
      scope: constant.language.boolean.false.js
      pop: true
    - match: null{{identifier_break}}
      scope: constant.language.null.js
      pop: true

  function-assignment:
    - match: |-
        (?x)(?=
          (?:{{identifier}} \s* \. \s*)*
          {{identifier}}
          \s* = \s*
          {{either_func_lookahead}}
        )
      set:
        - function-initializer
        - function-declaration-identifiers

  function-declaration-identifiers:
    - match: (?={{identifier}}\s*\.)
      push:
        - expect-dot-accessor
        - function-declaration-identifiers-expect-class
    - match: prototype{{identifier_break}}
      scope: support.constant.prototype.js
      pop: true
    - match: (?=#?{{identifier}})
      set:
        - function-name-meta
        - literal-variable-base

  expect-dot-accessor:
    - match: '{{dot_accessor}}'
      scope: punctuation.accessor.js
      pop: true
    - include: else-pop

  function-declaration-identifiers-expect-class:
    - match: prototype{{identifier_break}}
      scope: support.constant.prototype.js
      pop: true
    - include: language-identifiers
    - match: '{{dollar_only_identifier}}'
      scope: support.class.dollar.only.js punctuation.dollar.js
      pop: true
    - match: '{{dollar_identifier}}'
      scope: support.class.dollar.js
      captures:
        1: punctuation.dollar.js
      pop: true
    - match: '{{identifier}}'
      scope: support.class.js
      pop: true
    - include: else-pop

  function-name-meta:
    - meta_include_prototype: false
    - meta_scope: entity.name.function.js
    - include: immediately-pop

  either-function-declaration:
    - match: (?={{func_lookahead}})
      set: function-declaration
    - match: (?={{arrow_func_lookahead}})
      set: arrow-function-declaration

  function-declaration:
    - meta_include_prototype: false
    - match: ''
      set:
        - function-meta
        - function-declaration-expect-body
        - function-declaration-meta
        - function-declaration-expect-parameters
        - function-declaration-expect-name
        - function-declaration-expect-generator-star
        - function-declaration-expect-function-keyword
        - function-declaration-expect-async

  function-declaration-expect-body:
    - include: function-block
    - include: else-pop

  function-meta:
    - meta_include_prototype: false
    - meta_scope: meta.function.js
    - include: immediately-pop

  function-declaration-meta:
    - meta_include_prototype: false
    - meta_scope: meta.function.declaration.js
    - clear_scopes: 1
    - include: immediately-pop

  function-declaration-meta-no-clear:
    - meta_include_prototype: false
    - meta_scope: meta.function.declaration.js
    - include: immediately-pop

  function-declaration-expect-parameters:
    - include: function-declaration-parameters
    - include: else-pop

  function-declaration-expect-name:
    - match: '{{non_reserved_identifier}}'
      scope: entity.name.function.js
      pop: true
    - include: else-pop

  function-declaration-expect-generator-star:
    - match: \*
      scope: keyword.generator.asterisk.js
      pop: true
    - include: else-pop

  function-declaration-expect-function-keyword:
    - match: function{{identifier_break}}
      scope: storage.type.function.js
      pop: true
    - include: else-pop

  function-declaration-expect-async:
    - match: async{{identifier_break}}
      scope: storage.type.js
      pop: true
    - include: else-pop

  arrow-function-declaration:
    - meta_include_prototype: false
    - match: ''
      set:
        - function-meta
        - arrow-function-expect-body
        - function-declaration-meta
        - arrow-function-expect-arrow
        - arrow-function-expect-parameters
        - function-declaration-expect-async

  arrow-function-expect-body:
    - include: function-block
    - match: (?=\S)
      set:
        - block-meta
        - expression-no-comma

  arrow-function-expect-arrow:
    - match: =>
      scope: storage.type.function.arrow.js
      pop: true
    - include: else-pop

  arrow-function-expect-parameters:
    - match: '{{identifier}}'
      scope: variable.parameter.function.js
      pop: true
    - include: function-declaration-parameters
    - include: else-pop

  function-block:
    - match: \{
      scope: punctuation.section.block.begin.js
      set:
        - meta_scope: meta.block.js
        - match: \}
          scope: punctuation.section.block.end.js
          pop: true
        - include: statements

  function-declaration-parameters:
    - match: \(
      scope: punctuation.section.group.begin.js
      set:
        - match: \)
          scope: punctuation.section.group.end.js
          pop: true
        - include: function-parameter-binding-list

  label:
    - match: ({{identifier}})\s*(:)
      captures:
        1: entity.name.label.js
        2: punctuation.separator.js

  object-literal:
    - match: \{
      scope: punctuation.section.block.begin.js
      set: object-literal-contents

  object-literal-contents:
    - meta_scope: meta.mapping.js

    - match: \}
      scope: punctuation.section.block.end.js
      pop: true

    - match: \.\.\.
      scope: keyword.operator.spread.js
      push: expression-no-comma

    - match: >-
        (?x)(?=
          {{property_name}}\s*:
          (?: {{either_func_lookahead}} )
        )
      push:
        - either-function-declaration
        - function-declaration-meta-no-clear
        - object-literal-expect-colon
        - object-literal-meta-key
        - method-name

    - match: '{{method_lookahead}}'
      push: method-declaration

    - match: '{{identifier}}(?=\s*(?:[},]|$|//|/\*))'
      scope: variable.other.readwrite.js

    - match: (?=\[)
      push:
        - object-literal-meta-key
        - computed-property-name

    - match: (?=\"|')
      push:
        - object-literal-meta-key
        - literal-string

    - match: (?=[-+]?(?:\.[0-9]|0[bxo]|\d))
      push:
        - object-literal-meta-key
        - literal-number

    # - include: bare-property-name
    - match: (?={{identifier}})
      push:
        - object-literal-meta-key
        - bare-property-name

    - include: comma-separator
    - match: ':'
      scope: punctuation.separator.key-value.js
      push: expression-no-comma

    # In case we're inside a destructured arrow function parameter that we
    # misidentified as an object literal.
    - match: '='
      scope: keyword.operator.assignment.js
      push: expression-no-comma

  bare-property-name:
    - match: '{{identifier}}'
      pop: true

  computed-property-name:
    - match: \[
      scope: punctuation.section.brackets.begin.js
      set:
        - match: \]
          scope: punctuation.section.brackets.end.js
          pop: true
        - match: (?=\S)
          push: expression

  object-literal-meta-key:
    - meta_scope: meta.mapping.key.js
    - include: else-pop

  object-literal-expect-colon:
    - match: ':'
      scope: punctuation.separator.key-value.js
    - include: else-pop

  method-name:
    - match: '{{dollar_identifier}}'
      scope: meta.mapping.key.dollar.js entity.name.function.js
      captures:
        1: punctuation.dollar.js
      pop: true
    - match: '{{identifier}}'
      scope: entity.name.function.js
      pop: true
    - match: "'"
      scope: punctuation.definition.string.begin.js
      set:
        - meta_include_prototype: false
        - meta_scope: meta.string.js string.quoted.single.js
        - meta_content_scope: entity.name.function.js
        - match: \'
          scope: punctuation.definition.string.end.js
          pop: true
        - match: \n
          scope: invalid.illegal.newline.js
          pop: true
        - include: string-content
    - match: '"'
      scope: punctuation.definition.string.begin.js
      set:
        - meta_include_prototype: false
        - meta_scope: meta.string.js string.quoted.double.js
        - meta_content_scope: entity.name.function.js
        - match: \"
          scope: punctuation.definition.string.end.js
          pop: true
        - match: \n
          scope: invalid.illegal.newline.js
          pop: true
        - include: string-content

    - match: (?=\[)
      push: computed-property-name

    - include: else-pop

  field-name:
    - match: '{{dollar_identifier}}'
      scope: meta.mapping.key.dollar.js variable.other.readwrite.js
      captures:
        1: punctuation.dollar.js
      pop: true
    - match: '{{identifier}}'
      scope: variable.other.readwrite.js
      pop: true
    - match: "'"
      scope: punctuation.definition.string.begin.js
      set:
        - meta_include_prototype: false
        - meta_scope: meta.string.js string.quoted.single.js
        - meta_content_scope: variable.other.readwrite.js
        - match: \'
          scope: punctuation.definition.string.end.js
          pop: true
        - match: \n
          scope: invalid.illegal.newline.js
          pop: true
        - include: string-content
    - match: '"'
      scope: punctuation.definition.string.begin.js
      set:
        - meta_include_prototype: false
        - meta_scope: meta.string.js string.quoted.double.js
        - meta_content_scope: variable.other.readwrite.js
        - match: \"
          scope: punctuation.definition.string.end.js
          pop: true
        - match: \n
          scope: invalid.illegal.newline.js
          pop: true
        - include: string-content
    - match: (#)({{identifier}})
      captures:
        1: punctuation.definition.variable.js
        2: variable.other.readwrite.js

    - match: (?=\[)
      push: computed-property-name

    - include: else-pop

  method-declaration:
    - meta_include_prototype: false
    - match: ''
      set:
        - function-meta
        - function-declaration-expect-body
        - function-declaration-meta
        - function-declaration-expect-parameters
        - method-name
        - method-declaration-expect-prefix
        - function-declaration-expect-async

  method-declaration-expect-prefix:
    - match: \*
      scope: keyword.generator.asterisk.js
    - match: (?:get|set){{identifier_break}}(?!\s*\()
      scope: storage.type.accessor.js
    - include: else-pop

  parenthesized-expression:
    - match: \(
      scope: punctuation.section.group.begin.js
      set:
        - meta_scope: meta.group.js
        - match: \)
          scope: punctuation.section.group.end.js
          pop: true
        - match: (?=\S)
          push: expression

  function-call-arguments:
    - match: (\.\?)?(\()
      captures:
        1: punctuation.accessor.js
        2: punctuation.section.group.begin.js
      set:
        - meta_scope: meta.group.js
        - match: \)
          scope: punctuation.section.group.end.js
          pop: true
        - include: expression-list

  array-literal:
    - match: \[
      scope: punctuation.section.brackets.begin.js
      set:
        - meta_scope: meta.sequence.js
        - match: \]
          scope: punctuation.section.brackets.end.js
          pop: true
        - include: expression-list

  property-access:
    - match: (\.\?)?(\[)
      captures:
        1: punctuation.accessor.js
        2: punctuation.section.brackets.begin.js
      push:
        - meta_scope: meta.brackets.js
        - match: \]
          scope: punctuation.section.brackets.end.js
          pop: true
        - match: (?=\S)
          push: expression

    - match: \.(?:\?)?
      scope: punctuation.accessor.js
      push:
        - match: (?={{identifier}}\s*(?:\.\?)?\()
          set:
            - call-method-meta
            - function-call-arguments
            - call-path
            - object-property
        - include: object-property

  literal-number:
    # floats
    - match: |-
        (?x:
          # 1., 1.1, 1.1e1, 1.1e-1, 1.e1, 1.e-1 | 1e1, 1e-1
          {{dec_integer}} (?: (\.) {{dec_digit}}* {{dec_exponent}}? | {{dec_exponent}} )
          # .1, .1e1, .1e-1
          | (\.) {{dec_digit}}+ {{dec_exponent}}?
        ){{identifier_break}}
      scope: constant.numeric.float.decimal.js
      captures:
        1: punctuation.separator.decimal.js
        2: punctuation.separator.decimal.js
      pop: true

    # integers
    - match: 0{{dec_digit}}+{{identifier_break}}
      scope: constant.numeric.integer.octal.js invalid.deprecated.numeric.octal.js
      pop: true

    - match: (0[Xx]){{hex_digit}}*(n)?{{identifier_break}}
      scope: constant.numeric.integer.hexadecimal.js
      captures:
        1: punctuation.definition.numeric.base.js
        2: storage.type.numeric.js
      pop: true

    - match: (0[Oo]){{oct_digit}}*(n)?{{identifier_break}}
      scope: constant.numeric.integer.octal.js
      captures:
        1: punctuation.definition.numeric.base.js
        2: storage.type.numeric.js
      pop: true

    - match: (0[Bb]){{bin_digit}}*(n)?{{identifier_break}}
      scope: constant.numeric.integer.binary.js
      captures:
        1: punctuation.definition.numeric.base.js
        2: storage.type.numeric.js
      pop: true

    - match: '{{dec_integer}}(n|(?!\.)){{identifier_break}}'
      scope: constant.numeric.integer.decimal.js
      captures:
        1: storage.type.numeric.js
      pop: true

    # illegal numbers
    - match: 0[Xx]{{identifier_part}}+
      scope: invalid.illegal.numeric.hexadecimal.js
      pop: true

    - match: 0[Bb]{{identifier_part}}+
      scope: invalid.illegal.numeric.binary.js
      pop: true

    - match: 0{{identifier_part}}+
      scope: invalid.illegal.numeric.octal.js
      pop: true

    - match: '[1-9]{{identifier_part}}+(?:\.{{identifier_part}}*)?'
      scope: invalid.illegal.numeric.decimal.js
      pop: true

  literal-call:
    - match: (?={{identifier}}\s*(?:\.\?)?\()
      set:
        - call-function-meta
        - function-call-arguments
        - literal-variable

    - match: (?={{identifier}}\s*(?:\.\s*{{identifier}}\s*)+(?:\.\?)?\()
      set:
        - call-method-meta
        - function-call-arguments
        - call-path
        - literal-variable

  call-path:
    - match: '{{dot_accessor}}'
      scope: punctuation.accessor.js
      push: object-property
    - include: else-pop

  call-function-meta:
    - meta_include_prototype: false
    - meta_scope: meta.function-call.js
    - include: else-pop

  call-method-meta:
    - meta_include_prototype: false
    - meta_scope: meta.function-call.method.js
    - include: else-pop

  call-function-name:
    - match: '{{dollar_only_identifier}}'
      scope: variable.function.js variable.other.dollar.only.js punctuation.dollar.js
      pop: true
    - match: '{{identifier}}'
      scope: variable.function.js
      pop: true
    - include: else-pop

  call-method-name:
    - include: support-property
    - match: '{{identifier}}'
      scope: variable.function.js
      pop: true
    - include: else-pop

  literal-variable:
    - include: language-identifiers
    - include: support

    - match: '{{constant_identifier}}(?=\s*[\[.])'
      scope: support.class.js
      pop: true

    - match: (?={{identifier}}\s*(?:\.\?)?\()
      set: call-function-name

    - include: literal-variable-base

  literal-variable-base:
    - match: '{{dollar_only_identifier}}'
      scope: variable.other.dollar.only.js punctuation.dollar.js
      pop: true
    - match: '{{dollar_identifier}}'
      scope: variable.other.dollar.js
      captures:
        1: punctuation.dollar.js
      pop: true
    - match: '{{constant_identifier}}'
      scope: variable.other.constant.js
      pop: true
    - match: '{{identifier}}'
      scope: variable.other.readwrite.js
      pop: true
    - match: (#)({{identifier}})
      captures:
        1: punctuation.definition.variable.js
        2: variable.other.readwrite.js
      pop: true

  language-identifiers:
    - match: arguments{{identifier_break}}
      scope: variable.language.arguments.js
      pop: true
    - match: super{{identifier_break}}
      scope: variable.language.super.js
      pop: true
    - match: this{{identifier_break}}
      scope: variable.language.this.js
      pop: true
    - match: globalThis{{identifier_break}}
      scope: variable.language.global.js
      pop: true

    # These three are ordinary variables, not literals!
    - match: undefined{{identifier_break}}
      scope: constant.language.undefined.js
      pop: true
    - match: NaN{{identifier_break}}
      scope: constant.language.nan.js
      pop: true
    - match: Infinity{{identifier_break}}
      scope: constant.language.infinity.js
      pop: true

  support:
    - include: support-variable-ecma
    - include: support-variable-console
    - include: support-variable-dom
    - include: support-variable-node

  support-variable-ecma:
    - match: Array{{identifier_break}}
      scope: support.class.builtin.js
      set:
        - match: '{{dot_accessor}}'
          scope: punctuation.accessor.js
          set:
            - include: support-property-ecma-array
            - include: object-property
            - include: else-pop
        - include: else-pop

    - match: ArrayBuffer{{identifier_break}}
      scope: support.class.builtin.js
      set:
        - match: '{{dot_accessor}}'
          scope: punctuation.accessor.js
          set:
            - include: support-property-ecma-arraybuffer
            - include: object-property
            - include: else-pop
        - include: else-pop

    - match: Atomics{{identifier_break}}
      scope: support.constant.builtin.js
      set:
        - match: '{{dot_accessor}}'
          scope: punctuation.accessor.js
          set:
            - include: support-property-ecma-atomics
            - include: object-property
            - include: else-pop
        - include: else-pop

    - match: BigInt{{identifier_break}}
      scope: support.class.builtin.js
      set:
        - match: '{{dot_accessor}}'
          scope: punctuation.accessor.js
          set:
            - include: support-property-ecma-bigint
            - include: object-property
            - include: else-pop
        - include: else-pop

    - match: Date{{identifier_break}}
      scope: support.class.builtin.js
      set:
        - match: '{{dot_accessor}}'
          scope: punctuation.accessor.js
          set:
            - include: support-property-ecma-date
            - include: object-property
            - include: else-pop
        - include: else-pop

    - match: JSON{{identifier_break}}
      scope: support.constant.builtin.js
      set:
        - match: '{{dot_accessor}}'
          scope: punctuation.accessor.js
          set:
            - include: support-property-ecma-json
            - include: object-property
            - include: else-pop
        - include: else-pop

    - match: Math{{identifier_break}}
      scope: support.constant.builtin.js
      set:
        - match: '{{dot_accessor}}'
          scope: punctuation.accessor.js
          set:
            - include: support-property-ecma-math
            - include: object-property
            - include: else-pop
        - include: else-pop

    - match: Number{{identifier_break}}
      scope: support.class.builtin.js
      set:
        - match: '{{dot_accessor}}'
          scope: punctuation.accessor.js
          set:
            - include: support-property-ecma-number
            - include: object-property
            - include: else-pop
        - include: else-pop

    - match: Object{{identifier_break}}
      scope: support.class.builtin.js
      set:
        - match: '{{dot_accessor}}'
          scope: punctuation.accessor.js
          set:
            - include: support-property-ecma-object
            - include: object-property
            - include: else-pop
        - include: else-pop

    - match: Promise{{identifier_break}}
      scope: support.class.builtin.js
      set:
        - match: '{{dot_accessor}}'
          scope: punctuation.accessor.js
          set:
            - include: support-property-ecma-promise
            - include: object-property
            - include: else-pop
        - include: else-pop

    - match: Proxy{{identifier_break}}
      scope: support.class.builtin.js
      set:
        - match: '{{dot_accessor}}'
          scope: punctuation.accessor.js
          set:
            - include: support-property-ecma-proxy
            - include: object-property
            - include: else-pop
        - include: else-pop

    - match: Reflect{{identifier_break}}
      scope: support.constant.builtin.js
      set:
        - match: '{{dot_accessor}}'
          scope: punctuation.accessor.js
          set:
            - include: support-property-ecma-reflect
            - include: object-property
            - include: else-pop
        - include: else-pop

    - match: String{{identifier_break}}
      scope: support.class.builtin.js
      set:
        - match: '{{dot_accessor}}'
          scope: punctuation.accessor.js
          set:
            - include: support-property-ecma-string
            - include: object-property
            - include: else-pop
        - include: else-pop

    - match: Symbol{{identifier_break}}
      scope: support.class.builtin.js
      set:
        - match: '{{dot_accessor}}'
          scope: punctuation.accessor.js
          set:
            - include: support-property-ecma-symbol
            - include: object-property
            - include: else-pop
        - include: else-pop

    - match: |-
        (?x:
          (?:
            BigInt64|
            BigUint64|
            Float(?:32|64)|
            Int(?:8|16|32)|
            Uint(?:8|16|32|32Clamped)
          )
          Array{{identifier_break}}
        )
      scope: support.class.builtin.js
      set:
        - match: '{{dot_accessor}}'
          scope: punctuation.accessor.js
          set:
            - include: support-property-ecma-typedarray
            - include: object-property
            - include: else-pop
        - include: else-pop

    # Classes with no constructor properties
    - match: (?:Boolean|DataView|Function|Map|RegExp|Set|WeakMap|WeakSet){{identifier_break}}
      scope: support.class.builtin.js
      pop: true
    - match: (?:Eval|Range|Reference|Syntax|Type|URI)?Error{{identifier_break}}
      scope: support.class.builtin.js
      pop: true

    - match: (?:eval|isFinite|isNaN|parseFloat|parseInt|decodeURI|decodeURIComponent|encodeURI|encodeURIComponent){{identifier_break}}
      scope: support.function.js
      pop: true

  support-property-ecma-array:
    - match: (?:from|isArray|of){{identifier_break}}
      scope: support.function.builtin.js
      pop: true

  support-property-ecma-arraybuffer:
    - match: isView{{identifier_break}}
      scope: support.function.builtin.js
      pop: true

  support-property-ecma-atomics:
    - match: (?:and|add|compareExchange|exchange|isLockFree|load|or|store|sub|wait|wake|xor){{identifier_break}}
      scope: support.function.builtin.js
      pop: true

  support-property-ecma-bigint:
    - match: (?:asUintN|asIntN){{identifier_break}}
      scope: support.function.builtin.js
      pop: true

  support-property-ecma-date:
    - match: (?:now|parse|UTC){{identifier_break}}
      scope: support.function.builtin.js
      pop: true

  support-property-ecma-json:
    - match: (?:parse|stringify){{identifier_break}}
      scope: support.function.builtin.js
      pop: true

  support-property-ecma-math:
    - match: (?:E|LN10|LN2|LOG10E|LOG2E|PI|SQRT1_2|SQRT2){{identifier_break}}
      scope: support.constant.builtin.js
      pop: true
    - match: (?:abs|acos|acosh|asin|asin|atan|atanh|atan2|cbrt|ceil|clz32|cos|cosh|exp|expm1|floor|fround|hypot|imul|log|log1p|log10|log2|max|min|pow|random|round|sign|sin|sinh|sqrt|tan|tanh|trunc){{identifier_break}}
      scope: support.function.builtin.js
      pop: true

  support-property-ecma-number:
    - match: (?:EPSILON|MAX_SAFE_INTEGER|MAX_VALUE|MIN_SAFE_INTEGER|MIN_VALUE|NEGATIVE_INFINITY|POSITIVE_INFINITY){{identifier_break}}
      scope: support.constant.builtin.js
      pop: true
    - match: (?:isFinite|isInteger|isNaN|isSafeInteger|NaN|parseFloat|parseInt){{identifier_break}}
      scope: support.function.builtin.js
      pop: true

  support-property-ecma-object:
    - match: (?:assign|create|defineProperties|defineProperty|entries|freeze|fromEntries|getOwnPropertyDescriptors?|getOwnPropertyNames|getOwnPropertySymbols|getPrototypeOf|is|isExtensible|isFrozen|isSealed|keys|preventExtensions|seal|setPrototypeOf|values){{identifier_break}}
      scope: support.function.builtin.js
      pop: true

  support-property-ecma-promise:
    - match: (?:all|race|reject|resolve|allSettled|any){{identifier_break}}
      scope: support.function.builtin.js
      pop: true

  support-property-ecma-proxy:
    - match: revocable{{identifier_break}}
      scope: support.function.builtin.js
      pop: true

  support-property-ecma-reflect:
    - match: (?:apply|construct|defineProperty|deleteProperty|get|getOwnPropertyDescriptor|getPrototypeOf|has|isExtensible|ownKeys|preventExtensions|set|setPrototypeOf){{identifier_break}}
      scope: support.function.builtin.js
      pop: true

  support-property-ecma-string:
    - match: (?:fromCharCode|fromCodePoint|raw){{identifier_break}}
      scope: support.function.builtin.js
      pop: true

  support-property-ecma-symbol:
    - match: (?:asyncIterator|hasInstance|isConcatSpreadable|iterator|match|replace|search|species|split|toPrimitive|toStringTag|unscopeables){{identifier_break}}
      scope: support.constant.builtin.js
      pop: true
    - match: (?:for|keyFor){{identifier_break}}
      scope: support.function.builtin.js
      pop: true

  support-property-ecma-typedarray:
    - match: (?:BYTES_PER_ELEMENT){{identifier_break}}
      scope: support.constant.builtin.js
      pop: true

  support-variable-console:
    # https://console.spec.whatwg.org/
    - match: console{{identifier_break}}
      scope: support.type.object.console.js
      set:
        - match: '{{dot_accessor}}'
          scope: punctuation.accessor.js
          set: builtin-console-properties
        - include: else-pop

  support-variable-dom:
    - match: XMLHttpRequest{{identifier_break}}
      scope: support.class.dom.js
      pop: true
    - match: (?:document|window|navigator){{identifier_break}}
      scope: support.type.object.dom.js
      pop: true
    - match: (?:clearTimeout|setTimeout){{identifier_break}}
      scope: support.function.dom.js
      pop: true

  support-variable-node:
    - match: global{{identifier_break}}
      scope: support.type.object.node.js
      pop: true

    - match: Buffer{{identifier_break}}
      scope: support.class.node.js
      pop: true

    - match: process{{identifier_break}}
      scope: support.constant.node.js
      set:
        - match: '{{dot_accessor}}'
          scope: punctuation.accessor.js
          set:
            - include: support-property-node-process
            - include: object-property
            - include: else-pop
        - include: else-pop

    # Module-level variables
    - match: (?:__dirname|__filename|exports){{identifier_break}}
      scope: support.constant.node.js
      pop: true
    - match: module{{identifier_break}}
      scope: support.constant.node.js
      set:
        - match: '{{dot_accessor}}'
          scope: punctuation.accessor.js
          set:
            - include: support-property-node-module
            - include: object-property
            - include: else-pop
        - include: else-pop
    - match: require{{identifier_break}}
      scope: support.function.node.js
      pop: true

  support-property-node-process:
    - match: (?:arch|argv|argv0|channel|config|connected|debugPort|env|execArgv|execPath|exitCode|mainModule|noDeprecation|pid|platform|ppid|release|stderr|stdin|stdout|throwDeprecation|title|traceDeprecation|version|versions){{identifier_break}}
      scope: support.constant.node.js
      pop: true
    - match: (?:abort|chdir|cpuUsage|cwd|disconnect|dlopen|emitWarning|exit|getegid|geteuid|getgit|getgroups|getuid|hasUncaughtExceptionCaptureCallback|hrtime|initGroups|kill|memoryUsage|nextTick|send|setegid|seteuid|setgid|setgroups|setuid|hasUncaughtExceptionCaptureCallback|umask|uptime){{identifier_break}}
      scope: support.function.node.js
      pop: true

  support-property-node-module:
    - match: (?:children|exports|filename|id|loaded|parent|paths){{identifier_break}}
      scope: support.constant.node.js
      pop: true
    - match: require{{identifier_break}}
      scope: support.function.node.js
      pop: true

  builtin-console-properties:
    - match: (?:warn|info|log|error|time|timeEnd|assert|count|dir|group|groupCollapsed|groupEnd|profile|profileEnd|table|trace|timeStamp){{identifier_break}}
      scope: support.function.console.js
      pop: true
    - include: object-property

  object-property:
    - match: |-
        (?x)(?=
          {{identifier}}
          \s* = \s*
          {{either_func_lookahead}}
        )
      set:
        - function-initializer
        - function-name-meta
        - object-property-base

    - include: support-property

    - match: (?={{identifier}}\s*(?:\.\?)?\()
      set: call-method-name

    - include: object-property-base
    - include: else-pop

  object-property-base:
    - match: '{{dollar_only_identifier}}'
      scope: meta.property.object.dollar.only.js punctuation.dollar.js
      pop: true
    - match: '{{dollar_identifier}}'
      scope: meta.property.object.dollar.js
      captures:
        1: punctuation.dollar.js
      pop: true
    - match: '{{identifier}}'
      scope: meta.property.object.js
      pop: true
    - match: '{{identifier_part}}+{{identifier_break}}'
      scope: invalid.illegal.illegal-identifier.js
      pop: true
    - match: (#)({{identifier}})
      captures:
        1: punctuation.definition.variable.js
        2: meta.property.object.js
      pop: true

  support-property:
    - include: support-property-ecma

  support-property-ecma:
    - match: constructor{{identifier_break}}
      scope: variable.language.constructor.js
      pop: true
    - match: prototype{{identifier_break}}
      scope: support.constant.prototype.js
      pop: true

    - match: (?:hasOwnProperty|isPrototypeOf|propertyIsEnumerable|toLocaleString|toString|valueOf){{identifier_break}}
      scope: support.function.js
      pop: true

    # Annex B
    - match: __proto__{{identifier_break}}
      scope: invalid.deprecated.js variable.language.prototype.js
      pop: true
    - match: (?:__defineGetter__|__defineSetter__|__lookupGetter__){{identifier_break}}
      scope: invalid.deprecated.js support.function.js
      pop: true

  import-meta-expression:
    - match: import{{identifier_break}}
      scope: keyword.import.js
      set: import-expression-end

  import-expression-end:
    - match: (?=\()
      set: function-call-arguments
    - match: \.
      scope: punctuation.accessor.js
      set:
        - match: meta{{identifier_break}}
          scope: variable.language.import.js
          pop: true
        - include: object-property
    - include: else-pop
name: JS Custom - TypeScript

@rchl
Copy link

rchl commented May 28, 2020

Can you please edit your comment so that it doesn't take 2 minutes to scroll it? You can use this syntax to paste your content: https://gist.github.com/ericclemmons/b146fe5da72ca1f706b2ef72a20ac39d

@tsujp
Copy link

tsujp commented May 28, 2020

@rchl done, TIL.

@Thom1729
Copy link
Owner Author

Something odd is happening. According to your preferences, your only configurations are “Default” and “React” (plus the special embed configuration), so I'm not sure where that TypeScript.sublime-syntax is coming from. (The syntax definition you posted does not have any of the TypeScript functionality in it, which explains why it's not working as expected.)

It looks like TypeScript highlighting is enabled in your React configuration. Rebuilding syntaxes should get rid of the extra TypeScript.sublime-syntax file and ensure that the React configuration will handle TypeScript. If you've already rebuilt with the preferences you posted above, then I'm confused as to where that file is coming from. Is it at exactly Packages/User/JS Custom/Syntaxes/TypeScript.sublime-syntax?

Can you post your preferences again, just to be sure?

@Thom1729
Copy link
Owner Author

Thom1729 commented Jun 23, 2020 via email

@tsujp
Copy link

tsujp commented Jun 24, 2020

I believe that latest would also apply to the styled-components scope too then @michaelblyons? seen here: #93 (comment)

@michaelblyons
Copy link

@tsujp 🤷‍♂️ I do not know. I do not have code that looks like that.

@Thom1729
Copy link
Owner Author

v2.4.0-beta.3 is out supporting generic arguments in extends and implements clauses.

There's a new typescript.old_style_assertions option to enable <T>foo assertions. The interface there is not final, but it should work for now. I suppose that I could have the TypeScript extension check whether the JSX extension is enabled, but extensions weren't designed to interact in that way and it would be rather a kludge.

@Thom1729
Copy link
Owner Author

I'm working on the documentation. (I won't merge it until the end, since the front page is from master.) Here's the first draft of the TypeScript docs:

typescript: boolean (Beta)

Highlight TypeScript. TypeScript support is currently in beta, meaning that there may be bugs and that the exact behavior is subject to change.

By default, old-style type assertions (e.g. <T>foo) are not highlighted. You can enable them via the typescript.old_style_assertions option. As an example, these user preferences will provide configurations for both plain TypeScript and TypeScript with JSX:

{
    "configurations": {
        "TypeScript": {
            "file_extensions": [ "ts" ],
            "typescript": {
                "old_style_assertions": true
            },
        },
        "TypeScript (JSX)": {
            "file_extensions": [ "tsx" ],
            "typescript": true,
            "jsx": true
        }
    }
}

@Thom1729
Copy link
Owner Author

Microsoft's TypeScript syntax definition uses the scope source.ts instead of source.js. Is this a useful feature for e.g. tool compatibility?

@rchl
Copy link

rchl commented Jun 24, 2020

Yes. For language servers we have to be able to differentiate between typescript and javascript files and that is done now using the main scope of the syntax.

@michaelblyons
Copy link

Microsoft's TypeScript syntax definition uses the scope source.ts instead of source.js. Is this a useful feature for e.g. tool compatibility?

I'd say "yes." I made a PR on FileIcons to mark source.js.typescript as TypeScript, but mimicking the Microsoft base scope will obviate any need to do that for other packages that expect source.ts. This is particularly important since what I added depends on the JSC variant being named "TypeScript."

@predragnikolic
Copy link

predragnikolic commented Jun 24, 2020

You can configure the base scope with JSCustom with the scope config option.

this will change the base scope from source.js.typescript to source.ts

"TypeScript": {
            "file_extensions": [ "ts" ],
            "typescript": true,
            "scope": "source.ts",
        },
        "TypeScriptReact": {
            "file_extensions": [ "tsx" ],
            "typescript": true,
            "scope": "source.tsx",
            "jsx": true
        }

@Thom1729
Copy link
Owner Author

@rchl @michaelblyons Is changing the root scope sufficient or is it important that all of the individual scopes also end with .ts?

@michaelblyons
Copy link

Heh. For the things I've run into so far, @predragnikolic's nudge toward the README that I didn't read is entirely sufficient. 😁

@rchl
Copy link

rchl commented Jun 24, 2020

Is changing the root scope sufficient or is it important that all of the individual scopes also end with .ts?

Good question. For language servers at least, I believe we never really had to target individual scopes specifically so the suffix of those shouldn't matter. Maybe @rwols can confirm.

I've never even noticed that it's like that for both TS and default JS syntax...

For file icons, I believe only base scope matters.

@Thom1729
Copy link
Owner Author

PR #97 would allow replacing .js with .ts or .tsx anywhere in the syntax. If it does turn out to be needed, I'll have to add some tests/do cleanup.

@rwols
Copy link

rwols commented Jun 24, 2020

For language servers at least, I believe we never really had to target individual scopes specifically so the suffix of those shouldn't matter.

I can confirm only the base scope matters.

It doesn't matter for LSP what the base scope will be. You can set source.tsx | source.ts.react as document selector to get it to work with both Microsoft's tmLanguage file as well as Thom's syntaxes (at least, for ST4. For ST3 you have to fill in the plain old path to the syntax file).

@Thom1729
Copy link
Owner Author

v2.4.0-beta.4 is out with the following improvements:

  • Add default configurations for TypeScript, with and without JSX.
  • Fix a bug where JS Custom would try to rebuild a deleted configuration, causing an error.
  • When rebuild_js_custom_syntaxes is called with a versions argument, don't delete syntaxes that don't match versions.
  • Add separate tests for TypeScript with and without JSX.

@Thom1729
Copy link
Owner Author

Fun fact: in order to support function call type arguments, Microsoft's official TypeScript syntax uses the following regular expressions:

(?=(((([_$[:alpha:]][_$[:alnum:]]*)(\s*\??\.\s*(\#?[_$[:alpha:]][_$[:alnum:]]*))*)|(\??\.\s*\#?[_$[:alpha:]][_$[:alnum:]]*))|(?<=[\)]))\s*(?:(\?\.\s*)|(\!))?((<\s*(((keyof|infer|awaited|typeof|readonly)\s+)|(([_$[:alpha:]][_$[:alnum:]]*|(\{([^\{\}]|(\{[^\{\}]*\}))*\})|(\(([^\(\)]|(\([^\(\)]*\)))*\))|(\[([^\[\]]|(\[[^\[\]]*\]))*\])|(\'([^\'\\]|\\.)*\')|(\"([^\"\\]|\\.)*\")|(\`([^\`\\]|\\.)*\`))(?=\s*([\<\>\,\.\[]|=>|&(?!&)|\|(?!\|)))))([^<>\(]|(\(([^\(\)]|(\([^\(\)]*\)))*\))|(?<==)\>|\<\s*(((keyof|infer|awaited|typeof|readonly)\s+)|(([_$[:alpha:]][_$[:alnum:]]*|(\{([^\{\}]|(\{[^\{\}]*\}))*\})|(\(([^\(\)]|(\([^\(\)]*\)))*\))|(\[([^\[\]]|(\[[^\[\]]*\]))*\])|(\'([^\'\\]|\\.)*\')|(\"([^\"\\]|\\.)*\")|(\`([^\`\\]|\\.)*\`))(?=\s*([\<\>\,\.\[]|=>|&(?!&)|\|(?!\|)))))(([^<>\(]|(\(([^\(\)]|(\([^\(\)]*\)))*\))|(?<==)\>|\<\s*(((keyof|infer|awaited|typeof|readonly)\s+)|(([_$[:alpha:]][_$[:alnum:]]*|(\{([^\{\}]|(\{[^\{\}]*\}))*\})|(\(([^\(\)]|(\([^\(\)]*\)))*\))|(\[([^\[\]]|(\[[^\[\]]*\]))*\])|(\'([^\'\\]|\\.)*\')|(\"([^\"\\]|\\.)*\")|(\`([^\`\\]|\\.)*\`))(?=\s*([\<\>\,\.\[]|=>|&(?!&)|\|(?!\|)))))([^<>\(]|(\(([^\(\)]|(\([^\(\)]*\)))*\))|(?<==)\>)*(?<!=)\>))*(?<!=)\>)*(?<!=)>\s*)?\())
(?<=\))(?!(((([_$[:alpha:]][_$[:alnum:]]*)(\s*\??\.\s*(\#?[_$[:alpha:]][_$[:alnum:]]*))*)|(\??\.\s*\#?[_$[:alpha:]][_$[:alnum:]]*))|(?<=[\)]))\s*(?:(\?\.\s*)|(\!))?((<\s*(((keyof|infer|awaited|typeof|readonly)\s+)|(([_$[:alpha:]][_$[:alnum:]]*|(\{([^\{\}]|(\{[^\{\}]*\}))*\})|(\(([^\(\)]|(\([^\(\)]*\)))*\))|(\[([^\[\]]|(\[[^\[\]]*\]))*\])|(\'([^\'\\]|\\.)*\')|(\"([^\"\\]|\\.)*\")|(\`([^\`\\]|\\.)*\`))(?=\s*([\<\>\,\.\[]|=>|&(?!&)|\|(?!\|)))))([^<>\(]|(\(([^\(\)]|(\([^\(\)]*\)))*\))|(?<==)\>|\<\s*(((keyof|infer|awaited|typeof|readonly)\s+)|(([_$[:alpha:]][_$[:alnum:]]*|(\{([^\{\}]|(\{[^\{\}]*\}))*\})|(\(([^\(\)]|(\([^\(\)]*\)))*\))|(\[([^\[\]]|(\[[^\[\]]*\]))*\])|(\'([^\'\\]|\\.)*\')|(\"([^\"\\]|\\.)*\")|(\`([^\`\\]|\\.)*\`))(?=\s*([\<\>\,\.\[]|=>|&(?!&)|\|(?!\|)))))(([^<>\(]|(\(([^\(\)]|(\([^\(\)]*\)))*\))|(?<==)\>|\<\s*(((keyof|infer|awaited|typeof|readonly)\s+)|(([_$[:alpha:]][_$[:alnum:]]*|(\{([^\{\}]|(\{[^\{\}]*\}))*\})|(\(([^\(\)]|(\([^\(\)]*\)))*\))|(\[([^\[\]]|(\[[^\[\]]*\]))*\])|(\'([^\'\\]|\\.)*\')|(\"([^\"\\]|\\.)*\")|(\`([^\`\\]|\\.)*\`))(?=\s*([\<\>\,\.\[]|=>|&(?!&)|\|(?!\|)))))([^<>\(]|(\(([^\(\)]|(\([^\(\)]*\)))*\))|(?<==)\>)*(?<!=)\>))*(?<!=)\>)*(?<!=)>\s*)?\())

I was going to borrow Microsoft's lookaheads as a temporary measure before reimplementing with branching, but upon reflection, I think not.

@predragnikolic
Copy link

I wonder if the regular expression was written by a human?

@wbond
Copy link

wbond commented Jun 24, 2020

I’m glad we’ve got a more maintainable solution moving forward.

@Thom1729
Copy link
Owner Author

Yeah, it cannot be exaggerated how big a deal branching is for TypeScript parsing.

@tsujp
Copy link

tsujp commented Jun 25, 2020

Holy code batman.

@Thom1729
Copy link
Owner Author

v2.4.0 is out. I still need to fix #98 and some related issues, but once those are resolved I'll be opening a 3.0 branch that uses branching to implement the remaining TypeScript features.

@Thom1729
Copy link
Owner Author

There is now a sublime-4 branch based on the most recent core syntax. In addition, type arguments are supported in function calls and template tags. (At the moment, the function name/template tag is marked as a plain variable rather than function.)

@tsujp
Copy link

tsujp commented Jul 17, 2020

@Thom1729 I've noticed a drop in syntax highlighting for multi-line object inline-types when paired with function components in React.

Example code with syntax breaking after the first line:

export const FooComponent = ({
  some,  // syntax highlighting which was present before now breaks at this line and onwards
  list,
  of,
  props,
}: {
  some: string
  list: string
  of: boolean
  props: number
}) => {
  // not important
}

@Thom1729
Copy link
Owner Author

I've just published v3.0.0-alpha.2. The 3.0.0 branch requires a prerelease Sublime version, but it should provide much better arrow function detection.

@tsujp
Copy link

tsujp commented Jul 17, 2020

Fixed on those.

@Thom1729
Copy link
Owner Author

For context, without the new branching feature in the prereleases, Sublime's parser can't look ahead over line boundaries. This means that when it sees export const FooComponent = ({, it has to guess whether the opening paren is part of a parenthesized expression or an arrow function. When in doubt, it would always guess a parenthesized expression, and there is special logic later on so that when it saw the =>, it would realize its mistake and highlight the remainder as an arrow function. Presumably, there was some flaw with this logic and the colon confused it.

The prereleases have a new branching feature. When the parser sees the =>, it can backtrack and highlight the entire arrow function properly. This should allow us to identify arrow functions with 100% accuracy — any bugs that remain should be properly fixable, rather than resorting to arcane lookaheads and fallbacks.

@tsujp
Copy link

tsujp commented Jul 17, 2020

Very interesting, and nice work too to both you and the ST4 devs.

@Thom1729
Copy link
Owner Author

I'm closing this because the beta release should fix everything in the initial psot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

10 participants