single connection between node and controller

[pohly]

Let me toss out a thought how we might improve (?) the communication between the nodes and the central controller.

Right now, each node connects to the controller to register itself. It then keeps the connection open to ensure that it notices when it needs to re-register. The controller then also connects to each node when it wants to create or delete volumes.

We could reduce the number of TCP connections to one and avoid exposing anything on the node to the rest of the cluster as follows:

• the node establishes a TCP connection to the controller without doing that through gRPC
• the controller accepts that connection, again without gRPC
• for each established TCP connection, the controller creates a gRPC connection with a custom dialer that just uses the existing TCP connection
• the controller then queries the node through that connection, i.e. gRPC communication is always just from controller to node
• the node re-connects when it notices that the connection gets closed or stops receiving traffic (because that may indicate a half-open TCP connection)

There will be a bit of plumbing involved, but once that is in place, the higher level communication would be simpler.

[pohly]

Obsoleted by PR #838