Serialisation format for Bisq 2 (Misq)

[chimp1984]

As we need to decide soon for the serialisation format we are going to use in Bisq 2 (Misq) I wanted to share my research and discuss what we should use.

Requirements:

• Type safety
• Fast
• Small

Additional nice-to-have:

• RPC support
• Deterministic serialisation

Candidates:

• Protocol Buffers
• Cap'n Proto
• FlatBuffers
• TLV based custom format

Protocol Buffers:

Pro:

• Widely used
• Used inside Google, so gets top prio support
• Many implementations
• gRPC
• We are familiar with it from Bisq

Con:

• Can be slow for large data

Cap'n Proto

Pro:

• Zero-copy deserialization
• Fast

Con:

• Smaller dev and language support

FlatBuffers

Pro:

• Zero-copy deserialization
• Fast
• Supports also Protocol Buffers .proto format
• Support for gRPC (not sure how extensive)

Con:

• Seems to be developed mainly for file storage in trusted environment (gaming use cases), not for network (from https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html)
• Smaller dev and language support
• "FlatBuffers was definitely designed for languages like C++, so Java and Go may not be as good a fit. A big issue in Java for example is that it can't read UTF-8 strings in-place" (from Existential questions and insights google/flatbuffers#4385)

TLV based custom format:

Pro:

• Full control on byte level
• Can provide deterministic byte representation
• Used by LN (https://github.com/lightning/bolts/blob/master/01-messaging.md#type-length-value-format).

Con:

• High dev effort
• Risks for implementation bugs
• Risks for security issues

Comparison/discussions:

• https://capnprot o.org/news/2014-06-17-capnproto-flatbuffers-sbe.html
• https://speice.io/2019/09/binary-format-shootout.html
• Existential questions and insights google/flatbuffers#4385

Conclusion:

I think Protocol Buffers seems to be the best choice. It does not meet the performance of Cap'n Proto and FlatBuffers but the support for gRPC and the large dev community/support and language support are a big plus. Security considerations are also important specially if not the core language implementation is used. In case of Cap'n Proto and FlatBuffers this work is often done by external developers and there are less eyes on it. Protocol Buffers has achieved inside Google (as well as outside) a huge network effect which is hard to be challenged by the technically maybe superior concepts implemented in Cap'n Proto and FlatBuffers.

Deterministic serialisation is not guaranteed by none of those (beside a TLV based custom format) but it is in practice achieved when avoiding maps (at least in Protocol Buffers ). It could be managed also with a version freeze in case an updated version would cause issues. The requirement for deterministic data is also an exceptional case (DAO) and it could be dealt differently with it as well (would be anyway better but comes with more effort - e.g. use a custom serialisation format where this is a hard requirement).

I have not looked closer to Simple Binary Encoding. Seems to be very fast but comes with its own trade-offs. It is used in high frequency trading context, which does not really match our context.

Any feedback/comments?

[devinbileck]

Have you considered either Apache Thrift or Apache Avro?
I don't have experience with either so I cant speak to their pros and cons. But as they are popular/common, it should be worth at least investigating them.

[chimp1984]

I did not looked closer, but when searching for comparisons between Protocol Buffers and Thrift I stumbled over those:

• "Protobuf serialized objects are about 30% smaller than Thrift." [1]
• "Protobuf is faster when using "optimize_for = SPEED" configuration" - this is default now [1]

Others: https://old.floatingsun.net/articles/thrift-vs-protocol-buffers/index.html

Benchmarks: https://github.com/eishay/jvm-serializers/wiki/ (though benchmarks have to be taken with care, depends a lot of application context)

So overall it seems Protocol Buffers and Thrift are not that much far apart.

Another interesting one might be Kryo. But have not looked closer and seems a smallish project which comes with the mentioned security risks.

Apache Avro seems to be favouring dynamic typing:
"The main difference between Avro and Thrift is that Thrift is statically typed"

[1] https://stackoverflow.com/questions/69316/biggest-differences-of-thrift-vs-protocol-buffers

[ghubstan]

I have worked with thrift on a project that used it's own rpc service. The link @chimp1984 added [1] is a reasonable generalization of diffs between protobuf and thrift: https://stackoverflow.com/questions/69316/biggest-differences-of-thrift-vs-protocol-buffers#69374

[chimp1984]

One important consideration was missing in my initial post:
It is not clear yet how Bisq 2 will deal with the DAO/BSQ codebase, but likely it will be required to be integrated (needed for BSQ fees and for the trade protocol). As we use the raw bytes from protobuf serialisation as input for the hashes in the DAO state monitoring this would come with the dependency to stick with protobuf if we do not want to re-write all that. Sure we still could use protobuf only for that but having 2 serialisation stacks is not great.

[alvasw]

I looked at all the mentioned ideas. The most flexible approach would be to create a custom TLV format, but we do not have the required resources, as you said. From the others, Cap'n Proto looks the most promising, but I would still prefer Protocol Buffers. Deserialization bugs usually lead to arbitrary code execution. One of these in the wire format (network layer) would be disastrous. In my opinion, we should select Protocol Buffers even if it's not the fastest one. But please keep in mind that I am probably biased because I have only worked with Protocol Buffers so far.