Type Mapping Rules

Because MessagePack uses a schema-less, polymorphic type system, and Go is a strongly-typed language, any Go implementation of MessagePack serialization will have to make choices about how Go types map onto MessagePack types, and vice-versa. This document aims to explain the rules that msgp uses, and the justifications behind them.

Numerical Precision and Overflow

Rule 1: No Overflow

msgp always attempts to encode Go values in the smallest wire representation possible without any loss in numerical precision. For example, even though a Go int is 64 bits on 64-bit hardware, the encoding of int(5) is one byte on the wire, and it will still be 5 when decoded.

As a consequence of this rule, msgp will never let you decode a value that would overflow the object you are decoding into. For instance, if you use msgp.ReadInt16Bytes() or (*Reader).ReadInt16() to read out an integer value, the method will only succeed if the value of the integer is between math.MinInt16 and math.MaxInt16. For clarity's sake, here is the actual code for (*Reader).ReadInt16():

// ReadInt16 reads an int16 from the reader
func (m *Reader) ReadInt16() (i int16, err error) {
	var in int64
	in, err = m.ReadInt64()
	if in > math.MaxInt16 || in < math.MinInt16 {
		err = IntOverflow{Value: in, FailedBitsize: 16}
		return
	}
	i = int16(in)
	return
}

Note that all the methods that read int64 values will never return overflow errors, since MessagePack does not support integers wider than 64 bits.

Tip: Use int/uint or int64/uint64 when you cannot be sure of the magnitude of the encoded value.

Rule 2: No Loss of Precision

msgp will always encode a Go float32 as a 32-bit IEEE-754 float, and a float64 as a 64-bit IEEE-754 float.

When decoding, it is legal to decode a 32-bit float on the wire as a Go float64, but the opposite is illegal. This is to avoid the possibility of losing numerical precision.

Tip: Don't mix-and-match; pick either float32 or float64 and use it everywhere.

Rule 3: Sign Matters

msgp will not allow a value that is a uint on the wire to be decoded into an int, and vice-versa.

The justification behind this is to prevent applications from failing sporadically because one implementation encodes int values and the other decodes uint values, for example. Those types are not strictly compatible, and thus it is treated as a type error.

(This is unlike the floating-point conversion rules, as neither uint nor int is a strict sub- or super-set of the other.)

Tip: Use mostly signed integers.

Structs and Maps

Like JSON, MessagePack has no notion of strongly-typed data structures. msgp encodes Go struct objects as MessagePack maps. Decoding maps into Go structs can present some peculiar edge cases.

Rule 1: Keys are string-able

msgp does not support decoding maps with keys that are not "string-able" (either str or bin type, although str is preferred.)

You can still manually decode arbitrary maps with the primitives built into the library.

Rule 2: Map-to-Struct decoding is an Intersect

The generated implementations of msgp.Unmarshaler and msgp.Decodable decode the intersection of the map being decoded and the map represented by the struct. One of the most important consequences of this is that it is perfectly valid for a decode operation to not mutate the method receiver and return no error.

For example, let's assume we have the following type:

type Thing struct {
    Name  string  `msg:"name"`
    Value float64 `msg:"value"`
}

The following objects would all be legal to decode for the Thing type:

{} // the object is not mutated
{"name":"bob"} // only "name" is mutated
{"name":"bob","value":0.0} // both "name" and "value" are mutated
{"name":"bob","uncle":"joe"} // "name" is mutated; "uncle" is ignored

Users should take care to reset the values of objects that are repeatedly decoded in order to avoid conflating a previously decoded value with a new one.

Null

In Go, maps and slices can both have nil values. However, msgp will never decode a map or a slice as a nil value, instead encoding them as a zero-length map and a zero-length slice, respectively.

The only types that are encoded as MessagePack null are pointers and interface{}.

Decoding a null object into anything yields a TypeError.