Inconsistent binary packet ordering

[Totodore]

Describe the bug
I'll try to explain the issue I have been experiencing with socketioxide + socket.io-client. When sending two independent packets in a very tiny timeframe, they both arrive pretty much the same time at the server. This seems to be working, all packets arrive at the server and are handled just fine, but my guess is that the ack or any other packets sent back to the client may get messed up in order because they are sent by two parallel-running tokio threads. I'll try to visualize. This is socketioxide:

And this is Socket.io: Note that the packets are sent back exactly the same as they were received. (At least I have not had any issues with my previous Socket.io server in this regard)

Still, I am just guessing that this causes the problem. I did not yet do extensive research on this problem. The solution with msgpack is also described on the official Socket.io documentation here. This way, it does not really matter in what order the packets are sent and received, as one packet is equal one ethernet frame and neither side needs to wait until all packets have arrived, ultimately solving the underlying issue (although not the preferred way in most cases).

Pros: packets with binary content are sent as one single WebSocket frame (if the WebSocket connection is established)

To Reproduce
And equivalent error can be triggered with the following client/server

Expected behavior
Socketioxide should always send the binary packet adjacently to the first data packet.

Additional context
This issue is mainly a copy from this comment :
#231 (comment) (@FabianHummel)

[kelnos]

I'm running into this too (I think?), using socketioxide 0.8 (I'm using warp 0.3), acting as a server for the openHAB cloud connector (source).

The cloud connector responds to proxy requests from the socketio server by sending a responseHeader event, followed by one or more responseContentBinary events, terminated with a responseFinished event. When making a request, I usually see ordering (from the socketioxide server's perspective) like:

responseHeader
responseContentBinary
responseContentBinary
responseContentBinary
responseFinished
responseContentBinary

I'm not sure this is exactly the same problem: I don't think ACKs are involved at all (none of the events sent require ACKs), and it's the binary events themselves that are received by my application that are out of order, not the ACKs.

If I tell tokio to use the current_thread runtime, the problem goes away.

[Totodore]

The main problem comes from here:

socketioxide/engineioxide/src/socket.rs

Line 198 in 3ab5cd7

internal_tx: mpsc::Sender<Packet>,

. At insertion into the channel, when there is adjacent binary payload that it is needed, an atomic ordering is required.

My two main options for this are:

• Putting a Mutex here but it would be quite bad because most of the time it is single message packet that is sent and therefore it doesn't require any locking / ordering.
• Modifying the channel so that it takes a Vec<Packet> rather than a Packet. Adjacent binary packets would be grouped in each Vec. But most of the Vec<Packet> would be of size 1 and it would cause many new Vec alloc.

[kelnos]

Hm, maybe that's not the same problem I'm seeing then. If I understand this bit of code correctly, that's for packets that are being sent out from the socketioxide server to the client. For me, I'm seeing messages in the other direction that are out of order. Which, now that I read through the engineioxide/socketioxide code some more, I don't quite understand how that could be happening, as it looks like the event is handled and sent up to the application on the same thread as it's received on from the websocket, without any queuing. So maybe the fault is in my code...