Encoding `plog` Files

Encoding plog Files

The first actual transcoding example will write the data back out to another PLog file. This is useful in combination with filters, because one truncate a larger PLog into a smaller one, write out a new PLog with only chosen types, or subsample a limited range of records, to create a smaller and more specific PLog file from a larger one.

Truncation

➜  polysync-transcode ibeo.25.plog --slice :2 plog -o ibeo.2.plog
➜  polysync-transcode ibeo.2.plog --type ps_log_record
ps_log_record x2

The first command created a new file ibeo.2.plog that contains a duplicate of the header ps_log_header from ibeo.25.plog, plus its first two records. This is because the filter argument '--slice :2' selects the records from the beginning of the file and stops after the second one. This can be checked with a binary comparison:

➜  wc -c ibeo.2.plog
28216 ibeo.2.plog
➜  cmp ibeo.2.plog ibeo.25.plog --bytes 28216

The first command just computes how many bytes to compare (which is the entire file ibeo.2.plog) and the second one is a standard command line tool to check binary equality.

Model Diagnostics

Polysync-transcode provides valuable introspection tools to create and diagnose sensor data models.

➜  polysync-transcode ibeo.25.plog --slice 2 dump
log_record {
    index: 2
    size: 5510
    prev_size: 26840
    timestamp: 1464470621735352
    msg_header {
        ...
    } 
    ps_byte_array_msg {
        ...
    } 
    ibeo.header {
        magic: 0xaffec0c2
        ...
        data_type: 0x2280
        time: 0xdaf49729f8d76ab6
    } 
    raw: [ 70 6c 91 2 0 0 0 0 f8 ab 90 2 ... 0 0 88 57 2c 5c ff 7f 0 0 2 0 ] (5450 elements)
} 

Arg, what is up with that last field raw: ...? The first two records in the file have no raw fields, just fully decoded records of type ibeo.vehicle_state and ibeo.scan_data, respectively. (Hint: the transcoder will read raw bytes like this whenever it cannot determine a better model.) Diagnose the problem:

➜  polysync-transcode ibeo.25.plog --slice 2 dump --loglevel debug1
...
transcode[debug1]: loading descriptions from "../share/ibeo.toml"
transcode[debug1]: loading descriptions from "../share/sensor.toml"
transcode[debug1]: loading descriptions from "../share/core.toml"
...
transcode[verbose]: index: 2 size: 5510 prev_size: 26840 timestamp: 1464470621735352 
detector[debug1]: ps_byte_array_msg matched from parent "msg_header"
detector[debug1]: ibeo.header matched from parent "ps_byte_array_msg"
detector[debug1]: type not detected, returning raw sequence
log_record {
    ...

Turning on the debug1 loglevel shows us that the type descriptions are loaded from TOML files. While parsing record #2, we see the types ps_byte_array_msg, and then ibeo.header were "matched", then the "type was not detected" so we got the raw sequence instead of a decoded type.

What if this result is surprising, and you think that ibeo type 0x2280 should be fully decoded because the description actually exists? Check more detail:

➜  polysync-transcode ibeo.25.plog --slice 2 dump --loglevel debug2
...
detector[debug2]: ibeo.vehicle_state not matched: parent "msg_header" != "ibeo.header"
detector[debug2]: ibeo.scan_data not matched: parent "msg_header" != "ibeo.header"
detector[debug2]: ibeo.header not matched: parent "msg_header" != "ps_byte_array_msg"
detector[debug2]: ps_image_data_msg: mismatched { type: 16 != 9 }
detector[debug2]: ps_can_frame_msg: mismatched { type: 16 != 9 }
detector[debug1]: ps_byte_array_msg matched from parent "msg_header"
decoder[debug2]: decoding "ps_byte_array_msg" at offset 28256
...
detector[debug2]: ibeo.vehicle_state not matched: parent "ps_byte_array_msg" != "ibeo.header"
detector[debug2]: ibeo.scan_data not matched: parent "ps_byte_array_msg" != "ibeo.header"
detector[debug2]: ps_image_data_msg not matched: parent "ps_byte_array_msg" != "msg_header"
detector[debug2]: ps_byte_array_msg not matched: parent "ps_byte_array_msg" != "msg_header"
detector[debug2]: ps_can_frame_msg not matched: parent "ps_byte_array_msg" != "msg_header"
detector[debug1]: ibeo.header matched from parent "ps_byte_array_msg"
decoder[debug2]: decoding "ibeo.header" at offset 28272
...
detector[debug2]: ibeo.vehicle_state: mismatched { data_type: 0x2280 missing from description }
detector[debug2]: ibeo.scan_data: mismatched { data_type: 0x2280 missing from description }
detector[debug2]: ibeo.header not matched: parent "ibeo.header" != "ps_byte_array_msg"
detector[debug2]: ps_image_data_msg not matched: parent "ibeo.header" != "msg_header"
detector[debug2]: ps_byte_array_msg not matched: parent "ibeo.header" != "msg_header"
detector[debug2]: ps_can_frame_msg not matched: parent "ibeo.header" != "msg_header"
detector[debug1]: type not detected, returning raw sequence

Here, we see the transcoder searching for the type, including mismatched types. For more insight, the detectors are defined, partly in share/ibeo.toml. In there, the ibeo.header type contains two detectors: one for ibeo.vehicle_state and another for ibeo.scan_data. In the above debug output, the messages including not matched: parent are saying the message cannot match because the previous message, the "parent", is the wrong type. The other reason that a detector fails is because the field values do not match the (otherwise correct) parent. For instance, an ibeo.vehicle_state message always follows an ibeo.header, and for the match to succeed, two fields (magic and data_type), have to contain the exact Ibeo magic number 0xAFFEC0C2 and the exact data_type value 0x2807.

So now we see that the reason that record #2 fails to decode is because it has a field data_type with value 0x2280, which has no detector defined for it. To fix this, you would add a new detector to the ibeo.header section in the file ibeo.toml.