Json Parser - Parsing multiple values into Nested JSON

[JoeAshworth]

I'm trying to parse Envoy logs from a Docker container, however, after parsing the fields that start with the same name, I do not get nested JSON as I'd expect.

vector.yaml

sources:
  docker:
    type: docker_logs
    host_key: host
    include_containers:
      - envoy
transforms:
  json_parser:
    inputs:
      - docker
    type: json_parser
    field: message
sinks:
  console:
    type: console
    inputs:
      - geo
    target: stdout
    encoding: json

console output

{
    "container_created_at": "2022-01-27T14:38:47.063289449Z",
    "container_id": "83f1a2376b9a5b475xxxxxxxxxxxxxxxxxxx",
    "container_name": "envoy",
    "envoy.downstream_address": "172.xx.xxx.xxx",
    "envoy.request_duration": 0,
    "envoy.response_duration": 2,
    "envoy.upstream_address": "127.0.0.1:9901",
    "envoy.upstream_failure_reason": null,
    "host": "prelive-envoy-f2",
    "http.method": "GET",
    "http.path": "/metrics",
    "http.protocol": "HTTP/1.1",
    "http.request_duration": 0,
    "http.request_length": 0,
    "http.request_start_time": "2022-01-27T16:26:07.170Z",
    "http.response_code": 200,
    "http.response_code_details": "via_upstream",
    "http.response_duration": 359,
    "http.response_flags": "-",
    "http.response_length": 3152815,
    "http.total_duration": 361,
    "image": "envoyproxy/envoy:v1.20.1",
    "query_user_agent": "Prometheus/2.28.1",
    "source_type": "docker",
    "stream": "stdout",
    "timestamp": "2022-01-27T16:26:10.960218575Z"
}

I was expecting this output:

{
    "container_created_at": "2022-01-27T14:38:47.063289449Z",
    "container_id": "83f1a2376b9a5b475xxxxxxxxxxxxxxxxxxx",
    "container_name": "envoy",
    "envoy": {
      "downstream_address": "172.xx.xxx.xxx",
      "request_duration": 0,
      ...
    },
    "host": "prelive-envoy-f2",
    "http": {
      "method": "GET",
      "path": "/metrics",
      ...
    },
    "image": "envoyproxy/envoy:v1.20.1",
    "query_user_agent": "Prometheus/2.28.1",
    "source_type": "docker",
    "stream": "stdout",
    "timestamp": "2022-01-27T16:26:10.960218575Z"
}

Any suggestions on how I'd go about nesting those fields?

Many thanks!

[jszwedko]

Hi @JoeAshworth !

The json_parser transform will just parse JSON as it is. It seems like your input events have keys with .s in them. We are working on some changes to VRL that should permit this sort of transformation, but currently I think the only way you could do this is with custom Lua code via the lua transform.

[jszwedko]

This lua transform should work for you:

[transforms.nest]
type = "lua"
inputs = ["stdin"]
version = "2"
source = '''
function build_array(...)
  local arr = {}
  for v in ... do
    arr[#arr + 1] = v
  end
  return arr
end

function process(event, emit)
  for key, value in pairs(event.log) do
    event.log[key] = value

    parts = build_array(string.gmatch(key, "[^\\.]+"))

    -- key has a dot in it, break it apart to create nested table
    if #parts > 1 then
      event.log[key] = nil

      to_update = event.log
      for i = 1, #parts - 1 do
        if to_update[parts[i]] == nil then
          to_update[parts[i]] = {}
        end
        to_update = to_update[parts[i]]
      end
      to_update[parts[#parts]] = value
    end
  end
  emit(event)
end
'''
hooks.process = "process"

You'd want to put it right after the json_parser.

[JoeAshworth]

Many thanks @jszwedko!
That definitely would have taken me longer than an hour to learn Lua and get a working piece of code, much appreciated.

[jszwedko]

Glad it could help! I also opened https://github.com/vectordotdev/vector/issues/11066 to add a special purpose function for it to VRL given we do have the precedent of the flatten function which does the opposite of this.