Fix BufferedTokenizer to properly resume after a buffer full condition respecting the encoding of the input string#16968

Open

andsel wants to merge 9 commits intoelastic:mainfrom andsel:fix/buffered_tokenizer_clean_state_in_case_of_line_too_big_respecting_character_encoding

+394-17

Commits on Jan 31, 2025

Re-established existing tests with the addition of the encoding case transformation
andsel
committed
Fixed the test to compare Java strings byte rappresentation so that has the same encoding
andsel
committed
Re-imported previously reverted code
andsel
committed
Try to make the test reporduce the encoding problem
andsel
committed
Fixed test to verify the case of encoding preservation
andsel
committed
Switch from RubyArray addAll to concat method to preserve the encoding and avoid implicit deconding in addAll iterator
andsel
committed
Covered with more use cases and verifies encoding of the returned Ruby strings
andsel
committed
Updates the point of return Java String to return the one encoded with data input encoding, to do not change encoding
andsel
committed
Minor, removed unused import
andsel
committed