Destination to root of event

[pemontto]

Despite the documentation saying "The JSON format only supports simple key/value, unnested objects", the filter clearly supports nested objects and has done so for a while. It's even been suggested to use nested JSON objects in some github issues.

Given that it would be useful if the target could be set to the root, or some setting provided that would insert the nested fields straight into the root of the document. Otherwise it's quite a pain to grab the fields from the target field and move them to the root (unless there is an easy way I'm not sure of).

At the moment I have to store the JSON as a string inside the dictionary. Which requires the translate filter to look it up and the json filter to extract the fields.

• Version: 5.4.3
• Operating System: Centos 7
• Config File: nested.conf

input {
  stdin { codec => json }
}

filter {

  # Grab the user enrichment data
  translate {
    id => "proxy: lookup user"
    refresh_interval => 14400
    field => "user"
    exact => true
    dictionary_path => "loginid.json"
  }
}

output {
  stdout { codec => rubydebug }
}

• Sample Data:
  input lines:

{"user": "1"}
{"user": "2"}
{"user": "3"}

loginid.json

{
  "0": {
    "AccountIsDisabled": "False", 
    "DateCreated": "8/01/2015 5:59:17 PM"
  },
  "1": {
    "AccountIsDisabled": "False", 
    "DateCreated": "8/01/2015 5:59:17 PM"
  },
  "2": {
    "AccountIsDisabled": "False", 
    "DateCreated": "8/01/2015 5:59:17 PM"
  }
}

• Steps to Reproduce:
  ~ logstash -f nested.conf

{"user": "1"}
{
     "@timestamp" => 2017-06-30T14:32:25.604Z,
       "@version" => "1",
           "host" => "PerniciousMB.local",
    "translation" => {
        "AccountIsDisabled" => "False",
              "DateCreated" => "8/01/2015 5:59:17 PM"
    },
           "user" => "1"
}
{"user": "2"}
{
     "@timestamp" => 2017-06-30T14:32:28.364Z,
       "@version" => "1",
           "host" => "PerniciousMB.local",
    "translation" => {
        "AccountIsDisabled" => "False",
              "DateCreated" => "8/01/2015 5:59:17 PM"
    },
           "user" => "2"
}

[guyboertje]

As the fallback is a string and not a rich object, I think the safest path is to use an encoded string as the value and an encoded string as the fallback - meaning that an appropriate (JSON or Dissect) filter will always create keys and values in the Event root.

I think it would quite tricky to code for append to root only if JSON dictionary and the value is a map - well the code is not tricky but validating and documenting the specifics will be.