Resolve Split-Brain of WatchDocument with Stateful Session Affinity

[krapie]

Description:

Introduce stateful session affinity in yorkie cluster to resolve Split-Brain of WatchDocument. In other words, implementing session affinity to enable session routing with strong affinity over current ring hash with weak affinity.

Few months ago, envoy and istio have implemented stateful session affinity to resolve the same issue that we have. Below is how this stateful session affinity works in envoy & istio.

• If the session key x-session-key is empty in the request, envoy routes based on the existing LB policy of ring hash and returns x-session-key in the response with a base64 encoded value of the ip:port of the routed server to the client.
• Then, the client will set x-session-key header in the subsequent requests, allowing envoy to override the existing LB policy and route the request based on the decoded value ip:port.

One thing to keep in mind is that when backend host set changes, new client trying to attach to the document might be routed to different backend due to changes of ring hash calculation, similar to previous split-brain issue. But this time we know where the client should route to by x-session-key, so we can resolve this issue by storing x-session-key in "session store" like Redis or etcd, and provide the client on initialization (probably in AttachDocument).

To conclude, it's going to be a combination of both stateful session and ring hash. Stateful session will ensuring clients to connect to the same server on any situations. Ring hash will distribute the workloads(sessions) evenly with minimum disruption on backend host set changes.

Below are resources related to this issue.

• Server: Yorkie implementation branch with session affinity
• JavaScript SDK: Yorkie JS SDK branch with session affinity header
• Envoy documentation: Stateful session management in Envoy
• Utilizing etcd for the Session Store: Past yorkie server's etcd implementation
• Previous issues: Mitigate Split-brain of Long-lived Connection #526, Another approach to mitigate split-brain of a long-lived connection #625

Why:

By implementing session affinity, even after changes in the backend servers, clients will be routed to the same server for collaboration on shared documents. It ensures consistency for clients connecting to the same document.

This will completely resolve decreased sync sensitivity between peers caused by WatchDocument's split-brain issue.