From a2ae99999f51f7f7d9f868c97da40f62109e3db7 Mon Sep 17 00:00:00 2001 From: Daniel Armak Date: Thu, 26 Feb 2015 22:52:36 +0200 Subject: [PATCH 01/12] Update NETWORK_PROTOCOL.md --- NETWORK_PROTOCOL.md | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/NETWORK_PROTOCOL.md b/NETWORK_PROTOCOL.md index 2295c35..725adcf 100644 --- a/NETWORK_PROTOCOL.md +++ b/NETWORK_PROTOCOL.md @@ -1 +1,20 @@ -Placeholder for network protocol. +Status: this is a rough suggestion, just to get some ideas out there. + +Message framing will use protobuf. It's true that we don't need the full complexity of protobuf, but future protocol extensions might benefit. Also, an implementation might use a protobuf description of the published messages to decode both the framing and the messages in one go. + +Let a message frame = message type + size + contents. And let type Id = varint. The basic RS signalling would be: + + --> subscribe(publisher: Id, subscriber: Id, initialDemand: Long = 0) + --> request(subscriber: Id, demand: Long) + <-- subscribed(subscriber: Id) + <-- onNext(subscriber: Id, element: bytes) + <-- onComplete(subscriber: Id) + <-- onError(subscriber: Id, error: String) + +There is no separate Subscription object or id; the subscriber Id identifies the recipient in all messages going <-- this way. Each side must manage the mapping of Ids to publishers and subscribers. (Publisher Ids and subscriber Ids have separate namespaces.) + +The Ids are only large enough to identify components, not to describe them. So the discovery of publishers and their Ids must either happen out of band, or require more messages, which could form an optional protocol on top of this one (i.e. using other message type codes). The same is true for clients that ask the server to create parameterized publishers (e.g. to read a particular named file). + +The overhead per message is typically 1 byte (message code) + 1-2 bytes (recipient id) + 2-3 bytes (message length). When the message type is very small (e.g. an int), this is too much. There are three possible ways of handling this: + + 1. Send many messages at once, up to the current demand: `--> onNext(subscriber: Id, element: bytes, ...)`. The main problem with this is that the [WIP] From 63e575c58d37792ed68ae2f8708c3142d2e7985d Mon Sep 17 00:00:00 2001 From: Daniel Armak Date: Fri, 27 Feb 2015 00:19:43 +0200 Subject: [PATCH 02/12] WIP --- NETWORK_PROTOCOL.md | 61 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 55 insertions(+), 6 deletions(-) diff --git a/NETWORK_PROTOCOL.md b/NETWORK_PROTOCOL.md index 725adcf..5aea156 100644 --- a/NETWORK_PROTOCOL.md +++ b/NETWORK_PROTOCOL.md @@ -1,20 +1,69 @@ Status: this is a rough suggestion, just to get some ideas out there. -Message framing will use protobuf. It's true that we don't need the full complexity of protobuf, but future protocol extensions might benefit. Also, an implementation might use a protobuf description of the published messages to decode both the framing and the messages in one go. +## Message framing -Let a message frame = message type + size + contents. And let type Id = varint. The basic RS signalling would be: +Message framing will use protobuf. The full complexity of protobuf may not be needed today, but future protocol extensions might benefit. Also, an implementation might use a protobuf description of the published messages to decode both the framing and the messages using the same parser. - --> subscribe(publisher: Id, subscriber: Id, initialDemand: Long = 0) +A message frame begins with a message type, which is a varint, followed by its contents. Some messages are size-delimited, others are fixed-size. + +An `Id` is a varint. + +## Protocol negotiation + +The protocol is versioned and supports future extensions. The client (i.e. the side that opened the connection) and the server do a loose handshake: + + --> clientHello(version: varint, extensions: List[Id]) + <-- serverHello(version: varint, extensions: List[Id]) + +This is a 'loose' handshake because the server doesn't have to wait for the `clientHello` before sending its `serverHello`. + +The protocol version is currently version 0. If either side receives a hello message with a version it doesn't support, it MUST send a `goodbye` message and close the connection. The transport mapping (e.g. HTTP Content-Type, or TCP port number) SHOULD change in future versions when the protocol changes incompatibly and the version number increases. + +Extension to the protocol specify optional or future behaviors. + 1. If an extension defines a new message type not described in this specification, that message MUST NOT be sent before receiving a hello from the other side confirming that it supports that extension. + 2. If an extension changes the semantics of message types defined in this specification or by another extension, the modified behavior MUST be negotiated by at least one of the parties sending, and the other acknowledging, a message (defined by the extension being discussed) that declares the new behavior as active. A party supporting such an extension SHOULD NOT send messages whose semantics are modified by it before this negotiation is completed (i.e. the acknowledgement message is received). + +The client can optimistically send more messages after the `clientHello` without waiting for the `serverHello`. If it eventually receieves a `serverHello` with a different protocol version, it must consider that its messages were discarded. Future protocol versions will not be backward-compatible with version 0, in the sense that if a server multiple versions (e.g. both version 0 and some future version 1), it must wait for the `clientHello` and then send a `serverHello` with a version number matching the client's. + +## The Reactive Streams protocol + +Let type Id = varint. The basic RS signalling is: + + --> subscribe(publisher: String, subscriber: Id, initialDemand: Long = 0) --> request(subscriber: Id, demand: Long) + --> cancel(subscriber: Id) <-- subscribed(subscriber: Id) <-- onNext(subscriber: Id, element: bytes) <-- onComplete(subscriber: Id) <-- onError(subscriber: Id, error: String) + +The protocol is fully bidirectional; either party can act in the `-->` direction. + +The semantics for ordering and asynchronous delivery are the same as in the Reactive Streams specification. + +Unlike in RS, there is no separate Subscription object; the subscriber Id identifies the recipient in all messages going <-- this way. This id is generated by the subscriber and sent in the `subscribe` message. + +The publisher String needs to be parsed by the recipient; it is not described by this specification. [Could be added?] + +After a subscription is closed, its Id can be reused, to prevent Ids from growing without limit. The subscriber MAY reuse an Id in a `subscribe` message after it has sent `cancel` or received `onComplete` or `onError` for that Id. If it does so, it MUST guarantee that the publisher will not receive messages meant for the previous subscription with that Id after it receives the second `subscribe` message. + +## Closing the connection + +When the underlying transport is closed, both sides should release all related resources. This protocol version does not specify reusing previously negotiated state after reconnecting. + +The orderly way of closing a connection is to send a `goodbye` message, wait for acknowledgement and then close the underlying connection: + + --> goodbye(reason: String) + <-- goodbye(reason: String) + +Sending `goodbye` implicitly closes all open streams, equivalently to receiving `cancel` or `onError` messages. + +## Using lower-level multiplexing -There is no separate Subscription object or id; the subscriber Id identifies the recipient in all messages going <-- this way. Each side must manage the mapping of Ids to publishers and subscribers. (Publisher Ids and subscriber Ids have separate namespaces.) +## How to minimize overhead -The Ids are only large enough to identify components, not to describe them. So the discovery of publishers and their Ids must either happen out of band, or require more messages, which could form an optional protocol on top of this one (i.e. using other message type codes). The same is true for clients that ask the server to create parameterized publishers (e.g. to read a particular named file). +In typical use, the most common messages by far are `onNext`. The overhead per message is typically 1 byte (message code) + 1-2 bytes (subscriber id) + 2-3 bytes (message length). When the message type is very small (e.g. an int), this overhead can be greater than the payload size. -The overhead per message is typically 1 byte (message code) + 1-2 bytes (recipient id) + 2-3 bytes (message length). When the message type is very small (e.g. an int), this is too much. There are three possible ways of handling this: +There are several possible ways of handling this: 1. Send many messages at once, up to the current demand: `--> onNext(subscriber: Id, element: bytes, ...)`. The main problem with this is that the [WIP] From d486da6859b78aeb54bfa7fd0114f9623fabef45 Mon Sep 17 00:00:00 2001 From: Daniel Armak Date: Fri, 27 Feb 2015 00:43:17 +0200 Subject: [PATCH 03/12] WIP --- NETWORK_PROTOCOL.md | 39 ++++++++++++++++++++++----------------- 1 file changed, 22 insertions(+), 17 deletions(-) diff --git a/NETWORK_PROTOCOL.md b/NETWORK_PROTOCOL.md index 5aea156..deb946b 100644 --- a/NETWORK_PROTOCOL.md +++ b/NETWORK_PROTOCOL.md @@ -1,4 +1,4 @@ -Status: this is a rough suggestion, just to get some ideas out there. +Status: this is a rough draft, intended to get some ideas out there. ## Message framing @@ -6,10 +6,10 @@ Message framing will use protobuf. The full complexity of protobuf may not be ne A message frame begins with a message type, which is a varint, followed by its contents. Some messages are size-delimited, others are fixed-size. -An `Id` is a varint. - ## Protocol negotiation +An `Id` is a varint. + The protocol is versioned and supports future extensions. The client (i.e. the side that opened the connection) and the server do a loose handshake: --> clientHello(version: varint, extensions: List[Id]) @@ -32,21 +32,35 @@ Let type Id = varint. The basic RS signalling is: --> subscribe(publisher: String, subscriber: Id, initialDemand: Long = 0) --> request(subscriber: Id, demand: Long) --> cancel(subscriber: Id) - <-- subscribed(subscriber: Id) - <-- onNext(subscriber: Id, element: bytes) + <-- subscribed(subscriber: Id, elementSize: varint = 0) // For elementSize != 0, see the next section + <-- onNext(subscriber: Id, element: bytes) <-- onComplete(subscriber: Id) <-- onError(subscriber: Id, error: String) -The protocol is fully bidirectional; either party can act in the `-->` direction. - -The semantics for ordering and asynchronous delivery are the same as in the Reactive Streams specification. +The protocol is fully bidirectional; either party can act in the `-->` direction. The semantics for ordering and asynchronous delivery are the same as in the Reactive Streams specification. Unlike in RS, there is no separate Subscription object; the subscriber Id identifies the recipient in all messages going <-- this way. This id is generated by the subscriber and sent in the `subscribe` message. The publisher String needs to be parsed by the recipient; it is not described by this specification. [Could be added?] +The field `subscribed.elementSize`, if nonzero, indicates the fixed size of the elements that will be published in this stream. In fixed-size mode, the `onNext.element` field is not length-prefixed. This saves space when the messages are very small, such as individual ints. + After a subscription is closed, its Id can be reused, to prevent Ids from growing without limit. The subscriber MAY reuse an Id in a `subscribe` message after it has sent `cancel` or received `onComplete` or `onError` for that Id. If it does so, it MUST guarantee that the publisher will not receive messages meant for the previous subscription with that Id after it receives the second `subscribe` message. +## Packed messaging + +In typical use, the most common messages by far are `onNext`. The overhead per message is typically 1 byte (message code) + 1-2 bytes (subscriber id) + 2-3 bytes (message length) = 4-6 bytes total. When the message type is very small (e.g. an int), this overhead can be greater than the payload size. + +To reduce the overhead, the publisher can optionally declare that all stream elements will have a fixed size by setting the `subscribed.elementSize` field to a value greater than zero: + + <-- subscribed(subscriber: Id, elementSize: varint) + +The publisher can then send not just `onNext` messages but also `onNextPacked` messages: + + <-- onNextPacked(subscriber: Id, count: varint, messages: count * elementSize bytes) + +Packing does not help if new data becomes available very frequently and must not be delayed before being sent. A typical example is a ticker source. It also can't be done if the client doesn't provide enough demand. + ## Closing the connection When the underlying transport is closed, both sides should release all related resources. This protocol version does not specify reusing previously negotiated state after reconnecting. @@ -58,12 +72,3 @@ The orderly way of closing a connection is to send a `goodbye` message, wait for Sending `goodbye` implicitly closes all open streams, equivalently to receiving `cancel` or `onError` messages. -## Using lower-level multiplexing - -## How to minimize overhead - -In typical use, the most common messages by far are `onNext`. The overhead per message is typically 1 byte (message code) + 1-2 bytes (subscriber id) + 2-3 bytes (message length). When the message type is very small (e.g. an int), this overhead can be greater than the payload size. - -There are several possible ways of handling this: - - 1. Send many messages at once, up to the current demand: `--> onNext(subscriber: Id, element: bytes, ...)`. The main problem with this is that the [WIP] From 99b8204a436757ea173c5c6a84fd9865f9fdf704 Mon Sep 17 00:00:00 2001 From: Daniel Armak Date: Fri, 27 Feb 2015 00:58:52 +0200 Subject: [PATCH 04/12] First draft --- NETWORK_PROTOCOL.md | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/NETWORK_PROTOCOL.md b/NETWORK_PROTOCOL.md index deb946b..bd2af2c 100644 --- a/NETWORK_PROTOCOL.md +++ b/NETWORK_PROTOCOL.md @@ -1,5 +1,15 @@ Status: this is a rough draft, intended to get some ideas out there. +## Transport assumptions + +A supporting transport must be: + +1. Bidirectional and full-duplex +2. An octet stream, i.e. all octet values may be sent unencoded +3. Ordered delivery: an implementation may map protocol messages to some features of the underlying transport (e.g. 0mq messages), but the messages must arrive in the same order as they were sent + +Definitely supported transports include TCP, TLS (over TCP), WebSockets, and local pipes. HTTP/2 should be supported, but may require a dedicated specification for implementing this protocol. + ## Message framing Message framing will use protobuf. The full complexity of protobuf may not be needed today, but future protocol extensions might benefit. Also, an implementation might use a protobuf description of the published messages to decode both the framing and the messages using the same parser. @@ -25,7 +35,7 @@ Extension to the protocol specify optional or future behaviors. The client can optimistically send more messages after the `clientHello` without waiting for the `serverHello`. If it eventually receieves a `serverHello` with a different protocol version, it must consider that its messages were discarded. Future protocol versions will not be backward-compatible with version 0, in the sense that if a server multiple versions (e.g. both version 0 and some future version 1), it must wait for the `clientHello` and then send a `serverHello` with a version number matching the client's. -## The Reactive Streams protocol +## The Reactive Streams core protocol Let type Id = varint. The basic RS signalling is: @@ -61,6 +71,19 @@ The publisher can then send not just `onNext` messages but also `onNextPacked` m Packing does not help if new data becomes available very frequently and must not be delayed before being sent. A typical example is a ticker source. It also can't be done if the client doesn't provide enough demand. +## Split messaging + +A very large `onNext` message might significantly delay messages from other streams. Therefore, large stream elements can be optionally split across multiple messages. Publishers MAY split elements; subscribers MUST support this. + +When an element is split, the publisher will send one or more `onNextPart` messages, followed by a single `onNextLastPart`: + + <-- onNextPart(subscriber: Id, element: Id, data: bytes) + <-- onNextLastPart(subscriber: Id, element: Id, data: bytes) + +`element` is an Id assigned by the Publisher; messages with the same `element` value, in the same stream, will be joined by the Subscriber. The order of the parts is that in which they were sent and received (the transport is required to provide ordered delivery). + +The subscriber driver will typically join the parts transparently and deliver a single message to the application. + ## Closing the connection When the underlying transport is closed, both sides should release all related resources. This protocol version does not specify reusing previously negotiated state after reconnecting. @@ -71,4 +94,3 @@ The orderly way of closing a connection is to send a `goodbye` message, wait for <-- goodbye(reason: String) Sending `goodbye` implicitly closes all open streams, equivalently to receiving `cancel` or `onError` messages. - From e4b08b7be5db05efd2f1e4f9ec71b9badd8ca1a1 Mon Sep 17 00:00:00 2001 From: Daniel Armak Date: Fri, 27 Feb 2015 01:46:14 +0200 Subject: [PATCH 05/12] Added alternatives to protobuf --- NETWORK_PROTOCOL.md | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/NETWORK_PROTOCOL.md b/NETWORK_PROTOCOL.md index bd2af2c..0ba5342 100644 --- a/NETWORK_PROTOCOL.md +++ b/NETWORK_PROTOCOL.md @@ -1,33 +1,39 @@ Status: this is a rough draft, intended to get some ideas out there. +Missing: how to run on top of HTTP/2. Possibly the structure or semantics of publisher name strings. +Might be removed: extension support. +Need to choose serialization method (protobuf, msgpack, other?) + ## Transport assumptions -A supporting transport must be: +A supporting transport must be similar to a streaming socket: 1. Bidirectional and full-duplex 2. An octet stream, i.e. all octet values may be sent unencoded -3. Ordered delivery: an implementation may map protocol messages to some features of the underlying transport (e.g. 0mq messages), but the messages must arrive in the same order as they were sent +3. Ordered delivery: an implementation may map protocol messages to some features of the underlying transport (e.g. [Ømq](http://zeromq.org/) messages), but the messages must arrive in the same order as they were sent -Definitely supported transports include TCP, TLS (over TCP), WebSockets, and local pipes. HTTP/2 should be supported, but may require a dedicated specification for implementing this protocol. +Definitely supported transports include TCP, TLS (over TCP), WebSockets, and most socket-like objects (e.g. pipes). HTTP/2 will be supported, but may require a dedicated specification for implementing this protocol. ## Message framing -Message framing will use protobuf. The full complexity of protobuf may not be needed today, but future protocol extensions might benefit. Also, an implementation might use a protobuf description of the published messages to decode both the framing and the messages using the same parser. +An existing serialization format should be used. Current candidates are [Protocol Buffers](https://github.com/google/protobuf/) (which is slightly less space efficient), [MessagePack](http://msgpack.org/) (whose Scala implementation may not be as good as the others), and possibly [Thrift](https://thrift.apache.org/) (with which I'm less familiar). -A message frame begins with a message type, which is a varint, followed by its contents. Some messages are size-delimited, others are fixed-size. +The serialization format should be fast and space-efficient. It does not need to be self-describing, since the message types and their structures are fully specified in the protocol. It needs to have the types boolean, string / byte array (length-prefixed), and varint (an integer encoded using 1 or more bytes depending on its value). -## Protocol negotiation +The full complexity of these formats may not be needed today, but future protocol extensions might benefit. Also, an implementation might encode the published elements using the same format and decode both the framing and the messages using the same parser. -An `Id` is a varint. +Each message (frame) begins with a message type, which is a varint, followed by its contents. Messages are self-delimiting, because their structure is known from their type, and all fields are either of fixed size, self-delimited varints, or length-prefixed strings or byte arrays. + +## Protocol negotiation The protocol is versioned and supports future extensions. The client (i.e. the side that opened the connection) and the server do a loose handshake: - --> clientHello(version: varint, extensions: List[Id]) - <-- serverHello(version: varint, extensions: List[Id]) + --> clientHello(version: varint, extensions: Array[Id]) + <-- serverHello(version: varint, extensions: Array[Id]) This is a 'loose' handshake because the server doesn't have to wait for the `clientHello` before sending its `serverHello`. -The protocol version is currently version 0. If either side receives a hello message with a version it doesn't support, it MUST send a `goodbye` message and close the connection. The transport mapping (e.g. HTTP Content-Type, or TCP port number) SHOULD change in future versions when the protocol changes incompatibly and the version number increases. +The protocol version is currently version 0. If either side receives a hello message with a version it doesn't support, it MUST send a `goodbye` message (defined below) and close the connection. When future versions of the protocol introduce incompatible changes and increment the version number, transports SHOULD indicate the incompatibility when suitable, e.g. by changing the HTTP Content-Type or TCP port number). Extension to the protocol specify optional or future behaviors. 1. If an extension defines a new message type not described in this specification, that message MUST NOT be sent before receiving a hello from the other side confirming that it supports that extension. @@ -37,7 +43,7 @@ The client can optimistically send more messages after the `clientHello` without ## The Reactive Streams core protocol -Let type Id = varint. The basic RS signalling is: +The basic RS signalling is: --> subscribe(publisher: String, subscriber: Id, initialDemand: Long = 0) --> request(subscriber: Id, demand: Long) @@ -59,7 +65,7 @@ After a subscription is closed, its Id can be reused, to prevent Ids from growin ## Packed messaging -In typical use, the most common messages by far are `onNext`. The overhead per message is typically 1 byte (message code) + 1-2 bytes (subscriber id) + 2-3 bytes (message length) = 4-6 bytes total. When the message type is very small (e.g. an int), this overhead can be greater than the payload size. +In typical use, the most common messages by far are `onNext`. The overhead per message is typically 1 byte (message code) + 1-2 bytes (subscriber id) + 1-3 bytes (payload length) = 3-6 bytes total. When the message type is very small (e.g. an int), the overhead can be 100% or more. To reduce the overhead, the publisher can optionally declare that all stream elements will have a fixed size by setting the `subscribed.elementSize` field to a value greater than zero: @@ -82,7 +88,7 @@ When an element is split, the publisher will send one or more `onNextPart` messa `element` is an Id assigned by the Publisher; messages with the same `element` value, in the same stream, will be joined by the Subscriber. The order of the parts is that in which they were sent and received (the transport is required to provide ordered delivery). -The subscriber driver will typically join the parts transparently and deliver a single message to the application. +The subscriber's driver will typically join the parts transparently and deliver a single message to the application. ## Closing the connection From aa9c93564cc71360f1df0c62999a28dbabc1b97c Mon Sep 17 00:00:00 2001 From: Daniel Armak Date: Fri, 27 Feb 2015 11:16:02 +0200 Subject: [PATCH 06/12] Rename `subscribed` to `onSubscribe` to conform to RS --- NETWORK_PROTOCOL.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/NETWORK_PROTOCOL.md b/NETWORK_PROTOCOL.md index 0ba5342..ee34a42 100644 --- a/NETWORK_PROTOCOL.md +++ b/NETWORK_PROTOCOL.md @@ -48,7 +48,7 @@ The basic RS signalling is: --> subscribe(publisher: String, subscriber: Id, initialDemand: Long = 0) --> request(subscriber: Id, demand: Long) --> cancel(subscriber: Id) - <-- subscribed(subscriber: Id, elementSize: varint = 0) // For elementSize != 0, see the next section + <-- onSubscribe(subscriber: Id, elementSize: varint = 0) // For elementSize != 0, see the next section <-- onNext(subscriber: Id, element: bytes) <-- onComplete(subscriber: Id) <-- onError(subscriber: Id, error: String) @@ -59,7 +59,7 @@ Unlike in RS, there is no separate Subscription object; the subscriber Id identi The publisher String needs to be parsed by the recipient; it is not described by this specification. [Could be added?] -The field `subscribed.elementSize`, if nonzero, indicates the fixed size of the elements that will be published in this stream. In fixed-size mode, the `onNext.element` field is not length-prefixed. This saves space when the messages are very small, such as individual ints. +The field `onSubscribe.elementSize`, if nonzero, indicates the fixed size of the elements that will be published in this stream. In fixed-size mode, the `onNext.element` field is not length-prefixed. This saves space when the messages are very small, such as individual ints. After a subscription is closed, its Id can be reused, to prevent Ids from growing without limit. The subscriber MAY reuse an Id in a `subscribe` message after it has sent `cancel` or received `onComplete` or `onError` for that Id. If it does so, it MUST guarantee that the publisher will not receive messages meant for the previous subscription with that Id after it receives the second `subscribe` message. @@ -67,9 +67,9 @@ After a subscription is closed, its Id can be reused, to prevent Ids from growin In typical use, the most common messages by far are `onNext`. The overhead per message is typically 1 byte (message code) + 1-2 bytes (subscriber id) + 1-3 bytes (payload length) = 3-6 bytes total. When the message type is very small (e.g. an int), the overhead can be 100% or more. -To reduce the overhead, the publisher can optionally declare that all stream elements will have a fixed size by setting the `subscribed.elementSize` field to a value greater than zero: +To reduce the overhead, the publisher can optionally declare that all stream elements will have a fixed size by setting the `onSubscribe.elementSize` field to a value greater than zero: - <-- subscribed(subscriber: Id, elementSize: varint) + <-- onSubscribe(subscriber: Id, elementSize: varint) The publisher can then send not just `onNext` messages but also `onNextPacked` messages: From 866d51e166331caa79d54d44cf028263a489ec50 Mon Sep 17 00:00:00 2001 From: Daniel Armak Date: Tue, 3 Mar 2015 00:27:03 +0200 Subject: [PATCH 07/12] Change the message type and protocol version to a single Byte instead of a varint --- NETWORK_PROTOCOL.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/NETWORK_PROTOCOL.md b/NETWORK_PROTOCOL.md index ee34a42..59d5098 100644 --- a/NETWORK_PROTOCOL.md +++ b/NETWORK_PROTOCOL.md @@ -22,14 +22,14 @@ The serialization format should be fast and space-efficient. It does not need to The full complexity of these formats may not be needed today, but future protocol extensions might benefit. Also, an implementation might encode the published elements using the same format and decode both the framing and the messages using the same parser. -Each message (frame) begins with a message type, which is a varint, followed by its contents. Messages are self-delimiting, because their structure is known from their type, and all fields are either of fixed size, self-delimited varints, or length-prefixed strings or byte arrays. +Each message (frame) begins with a message type, which is a single byte, followed by its contents. Messages are self-delimiting, because their structure is known from their type, and all fields are either of fixed size, self-delimited varints, or length-prefixed strings or byte arrays. ## Protocol negotiation The protocol is versioned and supports future extensions. The client (i.e. the side that opened the connection) and the server do a loose handshake: - --> clientHello(version: varint, extensions: Array[Id]) - <-- serverHello(version: varint, extensions: Array[Id]) + --> clientHello(version: byte, extensions: Array[Id]) + <-- serverHello(version: byte, extensions: Array[Id]) This is a 'loose' handshake because the server doesn't have to wait for the `clientHello` before sending its `serverHello`. From 74382fa148a70fbcebe75e6a951be48e2916a5b0 Mon Sep 17 00:00:00 2001 From: Daniel Armak Date: Tue, 3 Mar 2015 00:30:33 +0200 Subject: [PATCH 08/12] Clarified types used This sentence was present in a previous draft but was accidentally deleted. --- NETWORK_PROTOCOL.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/NETWORK_PROTOCOL.md b/NETWORK_PROTOCOL.md index 59d5098..0629e97 100644 --- a/NETWORK_PROTOCOL.md +++ b/NETWORK_PROTOCOL.md @@ -18,7 +18,9 @@ Definitely supported transports include TCP, TLS (over TCP), WebSockets, and mos An existing serialization format should be used. Current candidates are [Protocol Buffers](https://github.com/google/protobuf/) (which is slightly less space efficient), [MessagePack](http://msgpack.org/) (whose Scala implementation may not be as good as the others), and possibly [Thrift](https://thrift.apache.org/) (with which I'm less familiar). -The serialization format should be fast and space-efficient. It does not need to be self-describing, since the message types and their structures are fully specified in the protocol. It needs to have the types boolean, string / byte array (length-prefixed), and varint (an integer encoded using 1 or more bytes depending on its value). +The serialization format should be fast and space-efficient. It does not need to be self-describing, since the message types and their structures are fully specified in the protocol. It needs to have the types boolean, byte, string / byte array (length-prefixed), and varint (an integer encoded using 1 or more bytes depending on its value). + +The type alias `Id` used below is a varint which serves to uniquely identify something. The full complexity of these formats may not be needed today, but future protocol extensions might benefit. Also, an implementation might encode the published elements using the same format and decode both the framing and the messages using the same parser. @@ -31,6 +33,8 @@ The protocol is versioned and supports future extensions. The client (i.e. the s --> clientHello(version: byte, extensions: Array[Id]) <-- serverHello(version: byte, extensions: Array[Id]) +An `Id`, as noted above, is a varint. An Array length-prefixed by a varint. + This is a 'loose' handshake because the server doesn't have to wait for the `clientHello` before sending its `serverHello`. The protocol version is currently version 0. If either side receives a hello message with a version it doesn't support, it MUST send a `goodbye` message (defined below) and close the connection. When future versions of the protocol introduce incompatible changes and increment the version number, transports SHOULD indicate the incompatibility when suitable, e.g. by changing the HTTP Content-Type or TCP port number). From a1bd203311edd7a351c146ae4638ac2622564992 Mon Sep 17 00:00:00 2001 From: Daniel Armak Date: Tue, 3 Mar 2015 00:46:49 +0200 Subject: [PATCH 09/12] Fix formatting of the first paragraph --- NETWORK_PROTOCOL.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/NETWORK_PROTOCOL.md b/NETWORK_PROTOCOL.md index 0629e97..55025e6 100644 --- a/NETWORK_PROTOCOL.md +++ b/NETWORK_PROTOCOL.md @@ -1,8 +1,11 @@ Status: this is a rough draft, intended to get some ideas out there. -Missing: how to run on top of HTTP/2. Possibly the structure or semantics of publisher name strings. -Might be removed: extension support. -Need to choose serialization method (protobuf, msgpack, other?) +Missing: +- How to run on top of HTTP/2 (but see #6). +- Possibly the structure or semantics of publisher name strings. +- Need to choose serialization method (protobuf, msgpack, other?) + +Possibly not needed and could be be removed: extension support. ## Transport assumptions From 8aebcaafb0b83cd666772224f27c1ed6b9590129 Mon Sep 17 00:00:00 2001 From: Daniel Armak Date: Tue, 3 Mar 2015 00:47:41 +0200 Subject: [PATCH 10/12] Add a note: could add feature to resume broken connections --- NETWORK_PROTOCOL.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/NETWORK_PROTOCOL.md b/NETWORK_PROTOCOL.md index 55025e6..7c3984c 100644 --- a/NETWORK_PROTOCOL.md +++ b/NETWORK_PROTOCOL.md @@ -1,9 +1,10 @@ Status: this is a rough draft, intended to get some ideas out there. -Missing: +Missing things which might be added: - How to run on top of HTTP/2 (but see #6). - Possibly the structure or semantics of publisher name strings. - Need to choose serialization method (protobuf, msgpack, other?) +- Support for resuming a broken connection without losing subscription state Possibly not needed and could be be removed: extension support. From 59a906423f266a9813f5a3f60e742a5b80f76ec3 Mon Sep 17 00:00:00 2001 From: Daniel Armak Date: Fri, 13 Mar 2015 22:50:24 +0200 Subject: [PATCH 11/12] Clarified extension negotiation --- NETWORK_PROTOCOL.md | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/NETWORK_PROTOCOL.md b/NETWORK_PROTOCOL.md index 7c3984c..d2c01b2 100644 --- a/NETWORK_PROTOCOL.md +++ b/NETWORK_PROTOCOL.md @@ -37,18 +37,24 @@ The protocol is versioned and supports future extensions. The client (i.e. the s --> clientHello(version: byte, extensions: Array[Id]) <-- serverHello(version: byte, extensions: Array[Id]) -An `Id`, as noted above, is a varint. An Array length-prefixed by a varint. +An `Id`, as noted above, is a varint. An Array is length-prefixed by a varint. This is a 'loose' handshake because the server doesn't have to wait for the `clientHello` before sending its `serverHello`. - + +### The protocol version + The protocol version is currently version 0. If either side receives a hello message with a version it doesn't support, it MUST send a `goodbye` message (defined below) and close the connection. When future versions of the protocol introduce incompatible changes and increment the version number, transports SHOULD indicate the incompatibility when suitable, e.g. by changing the HTTP Content-Type or TCP port number). -Extension to the protocol specify optional or future behaviors. - 1. If an extension defines a new message type not described in this specification, that message MUST NOT be sent before receiving a hello from the other side confirming that it supports that extension. - 2. If an extension changes the semantics of message types defined in this specification or by another extension, the modified behavior MUST be negotiated by at least one of the parties sending, and the other acknowledging, a message (defined by the extension being discussed) that declares the new behavior as active. A party supporting such an extension SHOULD NOT send messages whose semantics are modified by it before this negotiation is completed (i.e. the acknowledgement message is received). - The client can optimistically send more messages after the `clientHello` without waiting for the `serverHello`. If it eventually receieves a `serverHello` with a different protocol version, it must consider that its messages were discarded. Future protocol versions will not be backward-compatible with version 0, in the sense that if a server multiple versions (e.g. both version 0 and some future version 1), it must wait for the `clientHello` and then send a `serverHello` with a version number matching the client's. +### Protocol extensions + +Extensions allow for the protocol to be extended in the future in backward-compatible ways, without changing the protocol version. + + 1. The set of extensions in use, or available for use (for extensions that define optional behaviors), is the intersection of the extensions listed in both `hello` messages. + 2. Extensions MAY define new message types with new semantics. The client MUST NOT send messages of a new message type defined in an extension until it receives the `ServerHello` and confirms that the server supports the extension. + 3. Extensions MAY change the semantics of existing message types (e.g. to add transparent compression to payloads). Such modified behavior MUST be negotiated by one of the parties sending, and the other acknowledging, a message (defined by the extension being discussed) that declares the new behavior as active. A party supporting such an extension SHOULD NOT send messages whose semantics are modified by it before this secondary negotiation is completed, due to potential for confusion as to whether or not the modified semantics are in effect. + ## The Reactive Streams core protocol The basic RS signalling is: From 3514340f166c77ac008caaec0154559d065ddaf3 Mon Sep 17 00:00:00 2001 From: Daniel Armak Date: Fri, 13 Mar 2015 23:17:32 +0200 Subject: [PATCH 12/12] Reply to `goodbye` with `goodbyeAck` This makes it clear to both parties which party closed the session. --- NETWORK_PROTOCOL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/NETWORK_PROTOCOL.md b/NETWORK_PROTOCOL.md index d2c01b2..c558cbe 100644 --- a/NETWORK_PROTOCOL.md +++ b/NETWORK_PROTOCOL.md @@ -111,6 +111,6 @@ When the underlying transport is closed, both sides should release all related r The orderly way of closing a connection is to send a `goodbye` message, wait for acknowledgement and then close the underlying connection: --> goodbye(reason: String) - <-- goodbye(reason: String) + <-- goodbyeAck(message: String) Sending `goodbye` implicitly closes all open streams, equivalently to receiving `cancel` or `onError` messages.