FIX: Use lazy encoding in UTF-8 encoded string comparison

[milaGGL]

The previous fix created a performance issue due to expensive UTF-8 encoding. Update compareUtf8Strings to use lazy encoding instead.

[dconeybe]

Should this be left.length() - i?

[milaGGL]

why? do u mean encode all the remaining strings? 🤔

But we have confirmed that we encountered the characters that differ, which could be up to 2 index, considering surrogate pairs. If we are already promised that we can find the difference within 2 indexes, is there still a need to encode all remaining? for example, "ab👍cdefghjklm..." vs "ab👎ghjkiuytrd....".

I think substrings 2 indexes is already being very generous

[dconeybe]

Ahh you're right. I incorrectly thought left.length() was the number of chars in the substring. My bad. Nevermind this comment 🤦

[ehsannas]

Nit: @milaGGL I would improve the following comment to be more descriptive (along the lines of what you said above):

changing

// UTF-8 encoded byte comparison, substring 2 indexes to cover surrogate pairs

to

// We have identified a code point difference, so we don't need to
// encode/compare _all_ the remainder of the strings. We only need to 
// compare up to 2 more indexes to cover potential surrogate pairs.

[dconeybe]

This integration test coverage is good proof-of-concept. Please add solid unit test coverage too, as the logic is very delicate and exploits low-level properties of utf8 and utf16 encoding that are not well-known, obvious, or straight-forward.