Traffic Inspection Parcel
-------------------------

1. Introduction

The purpose of this document is to provide an engineering description of the Traffic Inspection Parcel (TIP) format so that it can be used and extended to describe high level network events.  TIP is based on a nested-TLV structure to provide an efficient, machine-readable format that can be easily parsed, extended, and segmented, and that can be used as either a file format or a serialization form for network transfer.

2. TIP Stream

Each top-level TLV is functionally an island unto itself.  Although more than one TLV may be necessary to tell a meaningful story of network events, each of these TLVs can be parsed, skipped, removed, or altered in isolation.  There is no strict ordering of these TLVs, nor is any particular type required to be present.  A TIP stream (for example, a single file) may contain several of these objects or merely one of them.

Furthermore, since each TLV makes its length externally-visible, they can be joined to one another arbitrarily without creating difficulties for the eventual parser of the stream.  For example, a legitimate (though pedestrian) merge of two TIP files can be performed by merely concatenating the contents of those files.

That said, generally each TLV is a piece of a larger puzzle. This freedom to segment, rearrange, and re-compose streams is a convenience only to the TIP parser - arbitrary mutations of a stream may likely deform the contained portrait of network events.  Reassemble these atomic pieces with care.

3. Top-Level Parcels

TIP is entirely composed, at the highest layer, of externally homogeneous TLVs called "parcels".  Each parcel has a 2-byte type field, which identifies the nature of its contents; a 4-byte "length" field, which describes the length of the contained content (up to 2^32 bytes, or 4gb); and an externally opaque "value" field exactly as long as the previously specified length.  All numeric fields, regardless of byte-width, are expressed as unsigned integers in network byte order.

The following is a real-life example of one complete top-level parcel and its contents:

+---- 2 ----+------- 4 -------+------- length = 9 bytes -------+
|   1A 01   |   00 00 00 09   |   4C 41 43 49 20 76 30 2e 39   |
+-----------+-----------------+--------------------------------+
  "made by"  length is 9 bytes           "LACI v0.9"

This particular parcel represents the TLV 3-tuple [0x1A01, 9, "LACI v0.9"].  In this case, a type of 0x1A01 is the "made by" parcel - it describes the software that was used to to generate this TIP stream.  The 6-bye length value declares that the content of this parcel is 9 bytes long, and correspondingly the content string, "LACI v0.9", is 9 bytes.  The length field does not include the width of the type or length fields, and zero is a valid length.

4. Common Data Structures

4.1 - Attribute Type (attr_type)

The attr_type is a single byte that describes the structure and the interpretation of the attribute.  There are three basic attribute types:  integer data, string, and TIP.

+---- 1 ----+
| attr type |
+-----------+

Integer data is described by types 0-63, binary + string data is described by types 64-127, and TIP-specific attributes are defined by types 128-191.  In other words, (attr_type LSB >> 6) == 0 if integer data, 1 if string data, and 2 if TIP data.  The following list specifies interpretation more precisely:
0x00 - This attribute is an unsigned integer.
0x01 - This attribute is an unsigned integer representing a boolean value: 0 == false/no, 1 == true/yes
0x02 - This attribute is an unsigned integer representing an IPv4 address (ie, 0x01020304 == 1.2.3.4)
0x03 - This attribute is an unsigned integer representing the number of nanoseconds elapsed since the Unix epoch
0x04 - This attribute is a signed integer where the first bit represents the sign (1 for negative, 0 for positive).
0x05 - This attribute is an unsigned integer code that can be converted into a string using the Attribute String Translator TIP.
0x40 - This attribute is a raw binary string.
0x41 - This attribute is a human-readable ascii string.
0x42 - This attribute is a 16-byte binary representation an IPv6 address (NBO).
0x43 - This attribute is an ascii string containing an error message.
0x80 - This attribute is a nested TIP list.
0x81 - This attribute is a nested TIP hash map.

4.2 - Attribute Description (attr_desc)

Attribute Description structures are used to describe the nature of an attribute within the defintion of an event or an instance of that event.  Attribute type structures are two bytes:

+------ 2 ------+
|   attr desc   |
+---------------+

The first (most significant) byte describes the presence of an attribute within an event.  The MSB values have the following meanings:

0x00 - This attribute may be present in this event or it may not be; it can appear either 0 or 1 times per event.
0x01 - This attribute is always present exactly once in this event.
0x02 - This attribute may appear 0, 1, or more times in this event.

The second byte (least significant) is always an attr_type.

4.3 - Event ID (event_id)

The Event ID is a two-byte field that uniquely identifies an event by its surrogate ID.

+------ 2 ------+
|   event id    |
+---------------+

These IDs can be mapped to human-readable names using the Event Dictionary parcel.

4.4 - Attribute ID (attr_id)

The Attribute ID is a two-byte field that uniquely identifies an attribute by its surrogate ID.

+------ 2 ------+
|    attr id    |
+---------------+

These IDs can be mapped to human-readable names using the Attribute Dictionary parcel.

5. Parcel Types

TIP is extensible in the sense that parcel types can be (responsibly!) added at any time without necessitating a change to the top-level TIP format.  Each type can have any format it wishes within its parcel, but all the default types continue with the TLV theme where appropriate.  Also, for ease of recognition, all default types start with a first byte of 0x1A.

At the time this document was last updated, these are the default types and their formats:

5.1 Made By (0x1A01)

Contains a string of data describing, in human-readable text form, the software that was used to generate this TIP stream.

5.2 Content Event (0x1ACE)

Content Event parcels describe one instance of an event.  Each instance follows the form of the definition of the event, described in the Event Structures parcel (if present).  In other words, the Content Event parcels describes event objects, where the Event Structures parcel describes event classes.

Contains a 2-byte "event id" followed by a concatenation of quasi-TLVs describing the event's attributes:

+------ 2 ------+
|   event id    |  +
+---------------+ 

+------ 2 ------+---- 1 ----+---------- 4 ----------+-- "length" --+
|    attr id    | attr type |  attr value's length  |  attr value  | ...
+---------------+-----------+-----------------------+--------------+

The "attr type" is a 1-byte field describing the nature of the provided value.  

String data is not allowed to include a terminating NULL character, and the consumer should never depend on it being present.

Binary data may either be human readable or not - no distinction is made.  NULL terminating characters (which are not part of the data to be communicated) are again not allowed.

TIP readers should only interpret the least significant byte of the "attr type" field, and TIP writers should always emit 0x00 as the most significant byte of the "attr type" field.

5.3 - Event Dictionary (0x1AED)

Contains a concatenation of sub-TLVs mapping 2-byte event IDs to their human-readable names:

+------ 2 ------+------ 2 ------+-- "length" --+
|   event id    |  name length  |  event name  | ...
+---------------+---------------+--------------+

Stylistically speaking, there is generally one event dictionary per TIP stream.  A TIP stream which possesses two or more contradictory dictionaries is not recommended and has undefined behavior.

5.4 - Attribute Dictionary (0x1AAD)

Contains a concatenation of sub-TLVs mapping 2-byte attribute IDs to their human-readable names:

+------ 2 ------+------ 2 ------+---- "length" ----+
|    attr id    |  name length  |  attribute name  | ...
+---------------+---------------+------------------+

Stylistically speaking, there is generally one attribute dictionary per TIP stream.  A TIP stream which possesses two or more contradictory dictionaries is not recommended and has undefined behavior.

5.5 - Attribute String Translator (0x1AA5)

Contains a 2-byte attribute ID followed by a concatenation of sub-TLVs mapping 4-byte, unsigned integer attribute values to their human readable string (eg, translating a DNS attribute QUESTION_TYPE value 15 into the more meaningful string "MX").  There is exactly one Attribute Translator parcel per numeric attribute requiring translation regardless of however many events use that attribute.  All events that share a translated attribute share the same translation mapping.

+---- 2 ----+     +------ 4 ------+---- 2 ----+---- "length" ----+
|  attr id  |  +  |  attr value   |  length   |  attribute text  | ...
+-----------+     +---------------+-----------+------------------+

5.6 - Attribute Characteristics (0x1AAC)

This parcel is used to describe the post-processing characteristics attribute values.  These characteristics can be combined with the OR operation to form a characteristic bitmap.  The characteristic codes are:

0x00000001 - This attribute is an annotation rather than a strict member of an event (eg, encapsulation metadata)
0x00000002 - This unsigned integer attribute can be converted to a string
0x00000004 - This attribute value is an IPv4 (uint) or IPv6 (string) address

The format of the data in this parcel is a concatenation of 6-byte entries:

+----- 2 -----+---------- 4 ----------+
|   attr id   | characteristic bitmap | ...
+-------------+-----------------------+

Integer attributes that can be converted to a string (0x00000008) are not necessarily convertible by the attribute string translator.  That is, some attribute values may be converted by the Network Content Inspection System (NCIS) itself that cannot reasonably be compiled to a static translation list.

5.7 - Event Structures (0x1AE5)

This parcel describes the structure of events that can be serialized in this TIP stream.  It is composed of a concatenation of sub-TLVs:

+------ 2 ------+------ 2 ------+     +------ 2 ------+------ 2 ------+ 
|   event id    | attrlist size |  +  |    attr id    |   attr desc   | ...
+---------------+---------------+     +---------------+---------------+

The "attrlist size" describes the size, in bytes, of the list of 4-byte [ attr_id, attr_type ] tuples.  That is, the number of distinct attributes that can be raised by the described event is "attrlist size" / 4. 

Example:  An attribute with an ID of 15 contains string data and will occur at most once per event.  The [ attr_id, attr_type ] tuple will get encoded as "\x00\x0F\x00\x20".
Example:  An attribute with an ID of 20 contains time_t data and may occur 0 or more times per event.  The [ attr_id, attr_type ] tuple will get encoded as "\x00\x14\x02\x03".

6 - Generic Data Structures in TIP

TIP can also serialize generic data structures into an attribute.

6.1 - List

This parcel merely contains a list of attributes, similar to an array.  The "list length" field describes the number of elements, not the number of bytes.

+------ 4 ------+     +---- 1 ----+------ 4 ------+---- "length" ----+
|  list length  |  +  | attr type |  attr length  |   attr content   | ...
+---------------+     +-----------+---------------+------------------+

6.2 - Hash Map

This parcel contains a hash map that associates attribute keys to attribute values.  The "hash length" describes the number of contained key/value pairs.

+------ 4 ------+ 
|  list length  |  +
+---------------+

   +---- 1 ----+------ 4 ------+---- "length" ----+
   | attr type |  attr length  |   attr content   | (key)  +
   +-----------+---------------+------------------+

   +---- 1 ----+------ 4 ------+---- "length" ----+
   | attr type |  attr length  |   attr content   | (value)  ...
   +-----------+---------------+------------------+

7 - Recommended Parcel Ordering

As described earlier, parcels can be written and read independently and do not formally depend on each other.  An event serialization system may only include Content Event parcels and ignore all the descriptive parcel types (they may in fact be redundant in some systems).

However, a "complete" TIP stream should generally contain all of the descriptive parcels followed by a series of Content Event parcels.  The recommended order is:

- Made By
- Event Dictionary
- Attribute Dictionary
- Attribute String Translator
- Attribute Characteristics
- Event Structures