Add binary option to WebSockets #1598
Conversation
SquidDev
added
the
area-Core
|
This should be passing tests, but it seems the in-game tests are randomly failing in areas I didn't touch - please disregard the failures. |
|
Thanks for the PR! Issues like this are always a pain, because the current behaviour is incorrect, and really sending/receiving raw UTF-8 is what we should have done from the start. I have mentioned this in a couple of places (#1246, #1233) and I am tempted to just remove text mode across the whole mod, always dealing with the raw bytes. Websockets are a little different, as they mandate UTF-8, but we can verify that when sending. Obviously this would break things, but I don't know how much. Maybe an experiment to run on SwitchCraft :).
Yeah, game tests are notoriously flaky in CI. I'm not quite sure why the windows one timed out - normally that's pretty reliable. I've kick-started both. |
| var buf = LuaValues.encode(text); | ||
| var bytes = new byte[buf.capacity()]; | ||
| buf.get(bytes); | ||
| data = new String(bytes, StandardCharsets.UTF_8); |
There was a problem hiding this comment.
It might be better to use a CharsetDecoder here instead, and then fail on CharacterCodingException errors, instead of replacing invalid characters with U+FFFD. Not sure, leave as is, will have a think about it.
Currently, handling of UTF-8 codes in WebSocket text messages is handled entirely on the Java side. This is understandable for most uses, but some users may want to interact with the bytes directly (e.g. with
utf8) without Java meddling with codepoints outside CC's range.This PR adds a binary flag to
http.websocketthat disables UTF-8 decoding of text messages on the Java side. When enabled, the binary flag of each message remains intact, but the data received will always be interpreted as binary data - no more?s replacing out-of-range codes. Sending text messages on a binary handle will cause Java to reinterpret the bytes as UTF-8 before sending, preserving manually-encoded codepoints. (This is an implementation detail, sinceTextWebSocketFrametakes a UTF-16String, so we need to convert it from UTF-8 ourselves.) If that fails for some reason, the sender will fall back to normal bytewise strings.The syntax for this flag matches
http.get's argument list, so there should be no confusion between the two lists, and no compatibility issues should arise - WebSockets still default to normal/text handles. I've added a test suite as well (which duplicates the original WS test) that ensures that UTF-8 bytes are preserved through the socket.