Netbeans cannot delete some text when editing TOML files

[niabot]

Apache NetBeans version

Apache NetBeans 24

What happened

When editing TOML files for Gradle projects the editor hangs if there is a temporary syntax error in the files. At first the editor won't let me type anything within unmatched curly braces. Deleting the braces is also not possible. Under some circumstances the entire editor (and netbeans) hangs (endless loop). I get the following or similar exceptions:

java.lang.IllegalStateException: Lexer org.netbeans.modules.languages.toml.TomlLexer@4eeb0fe6
  returned null token but lexerInput.readLength()=2
  lexer-state: -1
  tokenStartOffset=1778, readOffset=1780, lookaheadOffset=1781
  Chars: "\n\n" - these characters need to be tokenized.
Fix the lexer to not return null token in this state.
	at org.netbeans.lib.lexer.LexerInputOperation.checkLexerInputFinished(LexerInputOperation.java:435)
	at org.netbeans.lib.lexer.LexerInputOperation.nextToken(LexerInputOperation.java:193)
	at org.netbeans.lib.lexer.inc.IncTokenList.tokenOrEmbeddingImpl(IncTokenList.java:170)
	at org.netbeans.lib.lexer.inc.IncTokenList.tokenOrEmbedding(IncTokenList.java:165)
	at org.netbeans.api.lexer.TokenSequence.moveNext(TokenSequence.java:458)
	at org.netbeans.modules.textmate.lexer.SyntaxHighlighting$TSInfo.moveNextToken(SyntaxHighlighting.java:785)
	at org.netbeans.modules.textmate.lexer.SyntaxHighlighting$HSImpl.moveTheSequence(SyntaxHighlighting.java:482)
	at org.netbeans.modules.textmate.lexer.SyntaxHighlighting$HSImpl.moveNext(SyntaxHighlighting.java:431)
	at org.netbeans.modules.editor.lib2.highlighting.DirectMergeContainer$Wrapper.fetchNextHighlight(DirectMergeContainer.java:600)
	at org.netbeans.modules.editor.lib2.highlighting.DirectMergeContainer$HlSequence.updateMergeVars(DirectMergeContainer.java:337)
	at org.netbeans.modules.editor.lib2.highlighting.DirectMergeContainer$HlSequence.nextMerge(DirectMergeContainer.java:303)
	at org.netbeans.modules.editor.lib2.highlighting.DirectMergeContainer$HlSequence.moveNext(DirectMergeContainer.java:234)
	at org.netbeans.modules.editor.lib2.highlighting.HighlightsReader.readUntil(HighlightsReader.java:59)
	at org.netbeans.modules.editor.lib2.view.HighlightsViewFactory.restart(HighlightsViewFactory.java:168)
	at org.netbeans.modules.editor.lib2.view.ViewBuilder$FactoryState.init(ViewBuilder.java:1299)
	at org.netbeans.modules.editor.lib2.view.ViewBuilder.createViews(ViewBuilder.java:699)
	at org.netbeans.modules.editor.lib2.view.ViewBuilder.createReplaceRepaintViews(ViewBuilder.java:660)
	at org.netbeans.modules.editor.lib2.view.ViewUpdates.reinitAllViews(ViewUpdates.java:189)
	at org.netbeans.modules.editor.lib2.view.DocumentViewOp.checkViewsInited(DocumentViewOp.java:621)
	at org.netbeans.modules.editor.lib2.view.DocumentView.getPreferredSpan(DocumentView.java:226)
	at java.desktop/javax.swing.plaf.basic.BasicTextUI$RootView.getPreferredSpan(BasicTextUI.java:1454)
	at java.desktop/javax.swing.plaf.basic.BasicTextUI.getPreferredSize(BasicTextUI.java:950)
	at java.desktop/javax.swing.JComponent.getPreferredSize(JComponent.java:1734)
	at java.desktop/javax.swing.JEditorPane.getPreferredSize(JEditorPane.java:1390)
	at org.netbeans.modules.editor.NbEditorUI$LayeredEditorPane.getPreferredSize(NbEditorUI.java:454)
	at java.desktop/javax.swing.ScrollPaneLayout.layoutContainer(ScrollPaneLayout.java:799)
	at com.formdev.flatlaf.ui.FlatScrollPaneUI$FlatScrollPaneLayout.layoutContainer(FlatScrollPaneUI.java:563)
	at java.desktop/java.awt.Container.layout(Container.java:1541)
	at java.desktop/java.awt.Container.doLayout(Container.java:1530)
	at java.desktop/java.awt.Container.validateTree(Container.java:1725)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validate(Container.java:1660)
	at org.netbeans.core.windows.view.ui.DesktopImpl$LayeredLayout.layoutContainer(DesktopImpl.java:529)
	at java.desktop/java.awt.Container.layout(Container.java:1541)
	at java.desktop/java.awt.Container.doLayout(Container.java:1530)
	at java.desktop/java.awt.Container.validateTree(Container.java:1725)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validate(Container.java:1660)
	at java.desktop/javax.swing.RepaintManager$3.run(RepaintManager.java:757)
	at java.desktop/javax.swing.RepaintManager$3.run(RepaintManager.java:755)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:400)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:87)
	at java.desktop/javax.swing.RepaintManager.validateInvalidComponents(RepaintManager.java:754)
	at java.desktop/javax.swing.RepaintManager$ProcessingRunnable.run(RepaintManager.java:1896)
	at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:318)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:773)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:720)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:714)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:400)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:87)
	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:742)
	at org.netbeans.core.TimableEventQueue.dispatchEvent(TimableEventQueue.java:136)
[catch] at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)

Language / Project Type / NetBeans Component

Gradle multi-project

How to reproduce

Copy the following content in the editor (*.toml) and delete the last brace. It wont get deleted because of an exception. Worse can happen if the you add or delete other braces.

[plugins]
jooq-codegen = { id = "org.jooq.jooq-codegen-gradle", version.ref="jooq" }

Did this work correctly in an earlier version?

No / Don't know

Operating System

Windows 10

JDK

Java 21

Apache NetBeans packaging

Apache NetBeans binary zip

Anything else

No response

Are you willing to submit a pull request?

No

[matthiasblaesing]

Reproducible, @lkishalmi @vieiro could you please have a look?

[lkishalmi]

Checking.

[lkishalmi]

Could not reproduce. However there is something weird in the stacktrace above.

It has references of org.netbeans.modules.textmate.lexer.SyntaxHighlighting which should not be there.
Is it possibe that some TextMate grammar get registered in that mime-path?

[vieiro]

Hi, I can see what @niabot says. In my case NetBeans does not "hang", but can't delete the last } in the given example.

I'll try to take a look at it later on.

For the record, here's the message I get in the IDE Log:

SEVERE [global]
java.lang.IllegalStateException: Lexer org.netbeans.modules.languages.toml.TomlLexer@419edac2
  returned null token but lexerInput.readLength()=2
  lexer-state: -1
  tokenStartOffset=85, readOffset=87, lookaheadOffset=88
  Chars: "\n\n" - these characters need to be tokenized.
Fix the lexer to not return null token in this state.
	at org.netbeans.lib.lexer.LexerInputOperation.checkLexerInputFinished(LexerInputOperation.java:435)
	at org.netbeans.lib.lexer.LexerInputOperation.nextToken(LexerInputOperation.java:193)
	at org.netbeans.lib.lexer.inc.TokenListUpdater.relex(TokenListUpdater.java:602)
	at org.netbeans.lib.lexer.inc.TokenListUpdater.updateRegular(TokenListUpdater.java:255)
	at org.netbeans.lib.lexer.inc.TokenHierarchyUpdate$UpdateItem.update(TokenHierarchyUpdate.java:325)
	at org.netbeans.lib.lexer.inc.TokenHierarchyUpdate.processLevelInfos(TokenHierarchyUpdate.java:200)
	at org.netbeans.lib.lexer.inc.TokenHierarchyUpdate.updateImpl(TokenHierarchyUpdate.java:171)
	at org.netbeans.lib.lexer.inc.TokenHierarchyUpdate.update(TokenHierarchyUpdate.java:109)
	at org.netbeans.lib.lexer.TokenHierarchyOperation.textModified(TokenHierarchyOperation.java:585)
	at org.netbeans.spi.lexer.TokenHierarchyControl.textModified(TokenHierarchyControl.java:71)
	at org.netbeans.lib.lexer.inc.DocumentInput.textModified(DocumentInput.java:128)
	at org.netbeans.lib.lexer.inc.DocumentInput.removeUpdate(DocumentInput.java:121)
	at org.netbeans.lib.editor.util.swing.PriorityDocumentListenerList.removeUpdate(PriorityDocumentListenerList.java:91)
	at java.desktop/javax.swing.text.AbstractDocument.fireRemoveUpdate(AbstractDocument.java:285)
	at org.netbeans.editor.BaseDocument.fireRemoveUpdate(BaseDocument.java:1656)
	at org.netbeans.editor.BaseDocument.handleRemove(BaseDocument.java:1026)
	at org.netbeans.editor.BaseDocument$FilterBypassImpl.remove(BaseDocument.java:2625)
	at java.desktop/javax.swing.text.DocumentFilter.remove(DocumentFilter.java:84)
	at org.openide.text.CloneableEditorSupport$DocFilter.remove(CloneableEditorSupport.java:2361)
	at org.netbeans.editor.BaseDocument.remove(BaseDocument.java:939)
	at org.netbeans.editor.BaseKit$DeleteCharAction$3.run(BaseKit.java:2184)
	at org.netbeans.editor.GuardedDocument.runAtomicAsUser(GuardedDocument.java:333)
	at org.netbeans.editor.BaseKit$DeleteCharAction.actionPerformed(BaseKit.java:2176)
	at org.netbeans.modules.csl.core.CslEditorKit$GsfDeleteCharAction.actionPerformed(CslEditorKit.java:375)
	at org.netbeans.editor.BaseAction.actionPerformed(BaseAction.java:324)
	at java.desktop/javax.swing.SwingUtilities.notifyAction(SwingUtilities.java:1810)
	at java.desktop/javax.swing.JComponent.processKeyBinding(JComponent.java:2956)
	at java.desktop/javax.swing.JComponent.processKeyBindings(JComponent.java:3004)
	at java.desktop/javax.swing.JComponent.processKeyEvent(JComponent.java:2918)
	at java.desktop/java.awt.Component.processEvent(Component.java:6398)
	at java.desktop/java.awt.Container.processEvent(Container.java:2266)
	at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:4996)
	at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2324)
	at java.desktop/java.awt.Component.dispatchEvent(Component.java:4828)
	at java.desktop/java.awt.KeyboardFocusManager.redispatchEvent(KeyboardFocusManager.java:1952)
	at java.desktop/java.awt.DefaultKeyboardFocusManager.dispatchKeyEvent(DefaultKeyboardFocusManager.java:883)
	at java.desktop/java.awt.DefaultKeyboardFocusManager.preDispatchKeyEvent(DefaultKeyboardFocusManager.java:1146)
	at java.desktop/java.awt.DefaultKeyboardFocusManager.typeAheadAssertions(DefaultKeyboardFocusManager.java:1020)
	at java.desktop/java.awt.DefaultKeyboardFocusManager.dispatchEvent(DefaultKeyboardFocusManager.java:848)
	at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:4877)
	at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2324)
	at java.desktop/java.awt.Window.dispatchEventImpl(Window.java:2780)
	at java.desktop/java.awt.Component.dispatchEvent(Component.java:4828)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:775)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:720)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:714)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:400)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:87)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:98)
	at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:747)
	at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:745)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:400)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:87)
	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:744)
	at org.netbeans.core.TimableEventQueue.dispatchEvent(TimableEventQueue.java:136)
[catch] at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)
line 2:0 token recognition error at: '\n'
line 3:0 token recognition error at: '\n'

[vieiro]

I think this may happen in any other antlr4 lexers. The bridge hiccups when the lexer returns null for a token (due to a lexer/parsing error).

This does not mean that the antlr4 grammar is wrong. The antlr4 [lexer] grammar works correctly for correct files, but can't handle a file with a missing }, for example. For antlr4 grammars to work correctly with the antlr editor bridge, the grammars must contemplate an "error" token and not just return null on incorrect files.

[niabot]

@vieiro The simple example (deleting the last curly brace) did not "hang" Netbeans. But i also managed to get it stuck in an infinite loop by copying/cutting text. This was my workaround for not being able to type or delete a curly brace. This intial bug might have triggered another bug/effect down the line, but i have no stacktrace/exception of it.