Is there a way to "clean" invalid UTF-8 in a buffer?

[totalgee]

With an international keyboard, I can sometimes get an invalid UTF-8 code entered into the Zep buffer (e.g. if I type a ¨ without pressing the following (e) key to complete the ë). Then I get crashes because of the exception thrown in utf8::next() (throws invalid_code_point(cp)). I can catch this error, of course, but unless I "clean" the buffer somehow, it will just keep coming each time I call HandleKeyPress(). I'm looking into something like MultiByteToWideChar()/WideCharToMultiByte() round trip (on Windows, at least), with GetBufferText() and SetText() on the Zep Buffer, but I'm not sure if there's any nicer way you would recommend.

Once this character is in the buffer (or the command buffer, in Vim mode!), I can't get rid of it. At least I can move the cursor somewhere else, and I can even comment out the bad line, but I can't delete that line or those characters, for example.

Thanks,
Glen.

[totalgee]

(By the way, I don't care about UTF-8 in my particular application, but it can happen during live coding that I accidentally hit some key combination that gives me one of these in the buffer or Vim command queue, then I'm screwed... I just want it to be replaced by ? or whatever.)

[samhocevar]

I remember fixing a similar issue with Zep & Imgui. Out of curiosity, what is the value of cp for you? Because ¨ is not supposed to be an invalid codepoint.

[totalgee]

Sorry, I wrote the wrong thing in my original message. It actually doesn't throw invalid_code_point, it throws invalid_utf8 (the previous case in utf8::next(), with *it being (in this case) what looks like ý (y with an acute accent over it) -- the char value shows as -3 in the debugger, I think it's actually because what is sent to HandleKeyPress() is 65533 (0xFFFD) (which would be -3 as a 16-bit signed value). The err_code is INVALID_LEAD and cp is 0. As suggested in later messages, I should try filtering/handling this on my end, it sounds like. Thanks for all the suggestions.

[cmaughan]

My take is that we should filter these out up front, and understand why invalid characters are getting through. The UTF8 support is new and not well tested, but clearing the buffer sounds like a cludge instead of figuring out why it was corrupted initially.

[samhocevar]

Something that often happens on Windows is that surrogate pairs (for codepoints >= U+10000) are sent as two input events, one for the low surrogate character and one for the high surrogate. But these characters, taken individually, are indeed invalid Unicode characters. What the application input layer must do is store the surrogate character and wait until the other half is received before passing it to Zep.

Not sure if this is the exact problem, but if it can be of any help, you can see more information about how I fixed this in Imgui at ocornut/imgui#2815 . The function that stores the surrogate and waits for the next one is here: https://github.com/ocornut/imgui/pull/2815/files#diff-cee9beb740469eb5192dfe94aaee7320932590adade9d9b1d2ee392ac2e1e88cR1266

[samhocevar]

So, I suggest adding a test for io.InputQueueCharacters[n] == IM_UNICODE_CODEPOINT_INVALID at the end of include/zep/imgui/editor_imgui.h to skip invalid keyboard input.

[totalgee]

Thanks for all the suggestions, I will plan to handle this on my end (and thanks for the examples). By the way, I'm not sure if this will happen for others (likely not, because of the way you're handling/filtering these things before they get to Zep), but in my case I can repro it by doing this (with a Spanish keyboard):

• press i to be in Insert (Vim) mode
• press the ´ key (forward/acute accent) -- or could use the ¨ (umlaut) or something similar, like º or ª, which are all (plain or shifted) keys on the Spanish keyboard). Don't press any other key (e.g. to add an accent), instead immediately press the backspace key to delete.
• invalid_utf8 exception is thrown for the case: INVALID_LEAD

Again, I don't actually need/want accents in my code editor, but I do need to avoid the crash (or if I catch an exception, ensure the buffer is still in a usable state). It sounds like my best bet is to avoid sending this kind of input to Zep.