[Feature Request] official guide for wiring TextMate grammars

[souporserious]

Context

• This issue is not a bug report. (please use a different template for reporting a bug)
• This issue is not a duplicate of an existing issue. (please use the search to find existing issues)

Description

Hello, I've been struggling to find a solution that works with TextMate grammars and the latest releases for monaco-editor and monaco-editor-webpack-plugin. I've tried following the linked monaco-tm package and the issue here #1915, but it seems some of the theme helpers generateTokensCSSForColorMap, TokenizationRegistry, and Color are no longer available. I even read the wonderful blog post about the deep dive into how it's all working to try and better diagnose the issue and just can't seem to get any solution to work.

I previously had this working using monaco-editor-textmate, but that has recently broke between monaco-editor versions 0.29.1 and 0.30.0 zikaari/monaco-editor-textmate#26.

I've managed to distill the wiring of textmate down to the solution below based on monaco-editor-textmate and theia. I know it's not the most performant route per se, but I was hoping to use monaco as an enhancement on top of server-rendered shiki code blocks so this will be loaded on demand if someone wants to play around with code examples.

import type { Registry, StateStack } from 'vscode-textmate'
import { INITIAL } from 'vscode-textmate'
import * as monaco from 'monaco-editor'

/** Wires up monaco-editor with monaco-textmate */
export function wireTextMateGrammars(
  /** TmGrammar `Registry` this wiring should rely on to provide the grammars. */
  registry: Registry,

  /** `Map` of language ids (string) to TM names (string). */
  languages: Map<string, string>,

  /** The monaco editor instance to wire up. */
  editor: monaco.editor.ICodeEditor
) {
  const tokenTheme = editor['_themeService'].getColorTheme().tokenTheme
  const defaultForeground = tokenTheme._root._mainRule._foreground

  return Promise.all(
    Array.from(languages.keys()).map(async (languageId) => {
      const grammar = await registry.loadGrammar(languages.get(languageId))

      monaco.languages.setTokensProvider(languageId, {
        getInitialState: () => new TokenizerState(INITIAL),
        tokenize: (line: string, state: TokenizerState) => {
          const result = grammar.tokenizeLine(line, state.ruleStack)

          return {
            endState: new TokenizerState(result.ruleStack),
            tokens: result.tokens.map((token) => {
              const scopes = token.scopes.slice(0)

              for (let i = scopes.length - 1; i >= 0; i--) {
                const scope = scopes[i]
                const foreground = tokenTheme._match(scope)._foreground

                if (foreground !== defaultForeground) {
                  return {
                    ...token,
                    scopes: scope,
                  }
                }
              }

              return {
                ...token,
                scopes: scopes[scopes.length - 1],
              }
            }),
          }
        },
      })
    })
  )
}

class TokenizerState implements monaco.languages.IState {
  constructor(private _ruleStack: StateStack) {}

  public get ruleStack(): StateStack {
    return this._ruleStack
  }

  public clone(): TokenizerState {
    return new TokenizerState(this._ruleStack)
  }

  public equals(other: monaco.languages.IState): boolean {
    return (
      other instanceof TokenizerState &&
      (other === this || other.ruleStack === this.ruleStack)
    )
  }
}

For some reason between those versions of the editor, the matched foreground (tokenTheme._match(scope)._foreground) has changed and is now returning a different foreground in some cases causing the syntax highlighting to be off.

I was going to file as a bug, but since there is no official guide and these are accessing private APIs it makes sense they are prone to breaking. Is it possible this is something on the roadmap or in the meantime there is a solution to workaround this until official guides/utilities might be available? I appreciate any help and understand there are a lot of moving parts here to make this all work.

Monaco Editor Playground Link

No response

Monaco Editor Playground Code

No response