What is the idea behind offsets?

[Danaozhong]

When I was reading the spec about offsets, I initially interpreted in that you can specify the position of a field in the bitstream.
For example, consider the following zserio struct:

struct Offsets {
    uint32 u32Offset;

u32Offset:
    varint16 vi16OffsetArray[];
};

The way I interpreted it was that if I set u32Offset to 10, vi16OffsetArray would be arranged at the start of the 10th byte of the binary blob:

0-4 bytes: u32Offset
4-9 bytes: padding added due to the specified offset field.
10-end: vi16OffsetArray

What actually happens in the serialization code generates an assertion that double-checks that the byte position in the reader matches u32Offset. If the byte position mismatches u32Offset, an assertion is thrown.

0-4 bytes: u32Offset
4-end: vi16OffsetArray

In this scenario, u32Offset must/will be set to 4.

It seems to me that offsets can be used to retrieve the position of a field within the byte stream after de-serialization, as some sort of assertion. This makes me wonder what is the purpose of the offset variable?

• couldn't we omit storing the offset variable in the serialized data, as it is anyway calculated in initialize_offsets()?
• it seems that, besides the alignment, using offsets do not impact the binary data, assuming no assertion is thrown. Is that correct?
• it seems to me that offsets can be used as some sort of assertion to ensure the byte position of a field matches to what is expected. Is that the main purpose?

Thank you for your feedback, and please let me know if I overlooked or misunderstood something!

[mikir]

Thank you very much for such detailed description. Our documentation of offsets is very brief and I think that we should improve this. I don't think that you overlooked something.

• couldn't we omit storing the offset variable in the serialized data, as it is anyway calculated in initialize_offsets()?

Yes, you can omit storing the offset variable in the data. Actually, sometimes it is very difficult to calculate them. Therefore, the generated code offers initialize_offset() method which calculates them and store them directly in the data. This method can be called manually before write() or it is called automatically in serialize() method.

• it seems that, besides the alignment, using offsets do not impact the binary data, assuming no assertion is thrown. Is that correct?

Yes, this is correct.

• it seems to me that offsets can be used as some sort of assertion to ensure the byte position of a field matches to what is expected. Is that the main purpose?

The main purpose and probably the only meaningful use case of the offsets is during the custom deserialization. Consider the following example:

struct Header
{
    uint32 offset;
};

struct ExtraData
{
    uint64 extraData; // data which are huge and useless for some applications
};

struct Data
{
    uint64 data; // data which are really needed
};

struct Blob
{
    Header header;
    ExtraData extraData; // this can be skipped in some cases
header.offset:
    Data data;
};

Now, let's say we have whole Blob in binary stream (bits). As the first step, we can deserialize only Header. Then, let's suppose that we have application which does not need extra data at all. Then, we can skip whole deserialization of the extraData and jump directly to the data using the offset stored in header.offset.

So, in another words, we can do some optimization tricks during deserialization to skip deserialization of data which are not needed.

[wichert]

So it this intended to be a type-safe alternative to extern, where you delay or skip deserialising some fields?

[mikir]

So it this intended to be a type-safe alternative to extern, where you delay or skip deserialising some fields?

Actually, offsets were implemented before extern but yes, you can consider it as an alternative to extern from the deserialize point of view.