Parsing logs in JSON format

Introduction

How to parse logs in JSON format

Details

Logs can be formatted using JSON format and looks like this:

{
  "level": "TRACE",
  "logger": "logger0",
  "thread": "main0",
  "message": "some message0",
  "location": {
    "method": "method0",
    "file": "file0",
    "line": "0",
    "class": "a.Some0"
  },
  "tags": [
    "tag0"
  ],
  "hostname": "host0",
  "user": "root0",
  "@version": "1",
  "@timestamp": "2010-08-24T20:43:33.511+0000",
  "appId": "special_system0"
}
{
  "level": "DEBUG",
  "logger": "logger1",
  "thread": "main1",
  "message": "some message1",
  "location": {
    "method": "method1",
    "file": "file1",
    "line": "1",
    "class": "a.Some1"
  },
  "tags": [
    "tag1"
  ],
  "hostname": "host1",
  "user": "root1",
  "@version": "1",
  "@timestamp": "2011-08-24T20:43:33.511+0000",
  "appId": "special_system1"
}

To parse JSON format you have to define mapping from JSON XPath to log event property. In menu Tools select Show JSON pattern parser editor, paste your logs so you can define and test mappings.

For logs above format can be defined like this:

level=level
message=message
date=@timestamp
dateFormat=yyyy-MM-dd'T'HH\:mm\:ss.SSSZZZZ
thread=thread
file=location.file
class=location.class
method=location.method
line=location.line
logger=logger
ndc=ndc
note=note
markerColor=markerColor
mdcKeys=appId,user,hostname

Date format:

Specify the date format by adding the property 'dateFormat', using patterns from java.text.SimpleDateFormat. If your date is just millis from 01.01.1970 represented as long, type 'timestamp' as dateFormat.

Fields represented as MDC

Some fields from json like hostname, user can be parsed to log event as MDC. You have to define property 'mdcKeys' and list json 'xpaths' separated by ','

Custom levels

For custom level mapping, add the property 'customLevels' providing values in the format CustomLevel1=Log4jLevel,CustomLevel2=Log4jLevel. For example: customLevels=myInfo=INFO,MyWarn=WARN

Assumptions:

• File contain log events formatted as json.
• Json objects for log event have to separated by new line.
• Json object can be multi-line (pretty print)
• Line with json have to start with '{'