Ambiguous Unicode characters

[elivance]

VS Code added an update in 2021 to mitigate Trojan Source attacks, in which:

• Unicode characters that look visually similar but are in fact distinct, or
• Unicode directional formatting characters

... are used to write code which the compiler interprets in a different way to a human reader (ie. the human cannot distinguish similar glyphs, or the text editor displays different text order to the compiler's interpretation).

Helix currently appears to ignore directional formatting characters, but doesn't clearly highlight them, or make them identifiable. It doesn't highlight Unicode characters which could be easily confused.

Of course this shouldn't necessarily be the default behavior, but I think it would be valuable to add a feature to Helix which is similar to VS Code's solution: render the Unicode bytes inline (with a themed background) for invisible control characters, and highlight ambiguous Unicode characters.

I did see this issue where the Unicode replacement character (�) was mentioned for rendering control sequences #11255.

This is what Visual Studio displays (by default):

VS Code (from the release notes):

This is what Helix displays (with :set whitespace.render all):

See also:

• https://trojansource.codes/trojan-source.pdf
• Highlight confusable Unicode characters microsoft/vscode#136437
• Implements highlighting of unicode characters. microsoft/vscode#137508
• https://code.visualstudio.com/updates/v1_62
• https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-42574