How to encode the metadata of a tile

[mactrem]

A layer of COVTiles is basically divided into a metadata section and a FeatureTable with the actual vector data.
The FeatureTable has a colum oriented layout and uses a custom binary encoding.
The structure of a FeatureTable is described in the metadata section see for example the TypeScript decoder or the Protobuf schema. Currently the metadata section is encoded in a custom binary format.

However, using an existing file format to encode the metadata section would have advantages such as using an existing Interface Definition Language (IDL) to describe the structure of the metadata, easy implementation of encoder/decoder, or out-of-the-box support for versioning that a allows backwards and forwards compatibility of the metadata section. Other widley used state-of-the art file formats are also using the approach of combining a exisiting file format for encoding the metadata with a custom binary encoding of the actual data like for example ORC with ProtoBuf, Parquet with Thrift or Arrow with FlatBuffers.

Below is a brief comparison of the different formats with the currently used custom binary encoding:
Protobuf
Pros

• Integrated Interface Definition Language (IDL) for describing the structure of the metdata based on proto files
• Integrated support for versioning enables backward and forward compatible of the metadata section (schema evolution)
• Easy implementation of encoder/decoder

Cons

• Additional library for decoding the metadata section is needed which increases the bundle size ->but for example Mapbox's pbf library is only 3 KB in size
• (Small) increase in tile size mainly due to the additional tags (for versioning) compared to the currently used binary encoding
• Compatibility only works for metadata not for the actual data

FlatBuffers
Pros

• Same as Protobuf (IDL, versioning, convenient to use)
• Faster decoding of the metadata section as FlatBuffers is optimized for speed

Cons

• Same as Probobuf
• In addition as FlatBuffers is optimized for access to serialized data without parsing/unpacking a larger size compared to Protobuf basically because no Varint support, overhead based on alignment, ...

Currently I'm leaning toward replacing the binary encoding with one of the formats listed above, as the larger tile and library package size is rather negligible. Since COVTiles is optimized in terms of size, as its main use case is designed for the transfer over the network, I would prefer Protobuf over FlatBuffers.

[HarelM]

I would favor something that can be parsed easily without the need to write it in every language. i.e. I favor the use of protobuf as well.

[maxammann]

I think protobuf/flatbuffers (whether the former or latter depends on benchmarks of speed and size) has advantages for prototyping.

I think in the long term a custom format has strong advantages because it reduces the need of a complex runtime to decode tiles. Protobuf is quite a heavy dependency.

Profobuf generally shines in large systems with hundred and thousands of types where the IDL definition changes and is extended.

It depends ofc on the complexity, but it could make sense to build a custom format (maybe based on something like the postcard spec) to get the best performance out of it.

I feel like backwards compatibility is not that important when it comes to tiles because usually tiles are regenerated weekly/monthly.

[HarelM]

Using a proprietary format is extremely discouraged from my point of view, after maintaining a lot of systems and a lot of code.
Moreover when you talk about something that needs be read and write in multiple languages, maintaining this proprietary format is just a overhead you don't want to have. The more common software you use the less code you need to maintain and the faster you can go implementing new features and having a battle tested code that can work in a production environment.
If there is something extreme which requires a special handling, than maybe, but I've seen a lot of places where things were written from scratch instead of using something that works well, almost in all cases in the end, we replaces the code with common software.
The fact that we need to invent this new format of COV tiles is extreme on its own since the current formats are limited, but that doesn't mean every part of it should be new, the more we use common software the better we are off.
I can't stress this enough, if only the scares I carry could talk... :-)

[maxammann]

I generally agree with the argument of reducing maintenance cost and make translation to other language slightly easier.
Though the amount of code we are talking there about is probably between 50-100lines. Also, this is a file format which is at the very core of the whole ecosystem, and it is worth investing time and effort to find the most efficient binary representation of the above schema.

This is not a question of rolling your own cryptography.

Though this definitely needs to be benchmarked, if there is an advantage to create a custom implementation over choosing flatbuffers/protobuf. For a version 0.1 using something like flatbuffers or protobuf is totally fine though.

[mactrem]

That was my thought as well, to compare the custom implementation with Protobuf and see how both behave in terms of size and performance. I will write a protobuf implementation for the metadata section. Should be not to much effort.

[maxammann]

Sounds good! I wonder how difficult it is to implement the protobuf spec. So that means creating like 200-400 lines ones code to parse a protobuf message without any third party dependencies 🤔

Maybe worth an estimation after translting it to protobuf.