ll-analyzer or "Large Log files Analyzer" is a simple tool for helping Product Support Engineers to speed up incidents root cause analysis.
This tool origins is back in 2017, it came from the need of dealing with huge log files generated by the SAP Hybris commerce suite. However the ll-analyzer was created with the idea of being extensible to any kind of log structure.
What does it do?
In short:
- Load all files (logs) from the directory where is executed.
- Analyze concurrently those files looking for "issues"
- Summarize in reports all of the issues found.
Why ll-analyzer?
"Yes, we are in the cloud age but we still have to deal with On Prem customer"
Some of them with more than 50 nodes, which means dozens of logs with hundreds of thousands lines.
Then we realize that even with,
$ grep 'europeConsignmentExportJob' *.log > another_log.txt
$ grep 'Exception' *.log > yes_another_log.txt
:75498,75622w! chunk_of_log.txt
\|[0-9]{4,99}\sms\|statement
you will spend some time on those files.
# mac
env GOOS=darwin GOARCH=386 go build -o ll-analyzer
#windows
GOOS=windows GOARCH=amd64 go build -o ll-analyzer.exe
# linux
GOOS=linux GOARCH=amd64 go build -o ll-analyzer
- Create a directory logs or with number of the incident you are working with. (windows users Avoid the Desktop or another directory with subdirectories).
- Download or copy logs files to that directory.
- Open a terminal or window command line.
- Do a
export PATHto the ll-analyzer. On Windows and environment variable should be created. - just execute the tool.
- config.toml, is the configuration file where other kind of logs can be configured.
- console.tpl, is a the template file for console log reports
- jdbc.tpl, is a the template file for console log reports
Currently the tools provide analysis for two logs, console and jdbc. But, it can be extended to analyse other logs as well.
Why Go?
- Learn a new language
- Easily compilation and distribution of the tool between platforms
- Use of goroutines and channels (cheap threads)
How to implement a new log analyzer?
1.- Configuration. You define a set of regular expressions.
consolel logs
[[logs]]
logType = "console"
templateFile = "console.tpl"
reportFileName = "console-report.html"
separator = "|"
splitSeparator = "\n"
lineMatchRegex = '((?P<starting>^((STATUS \||ERROR \||INFO \|)( jvm 1 \|| wrapper \|)( main \|)))(?P<timestamp> \d{4}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2}.\d{3}))'
fieldsMatchRegex = [
['errorLine','((?P<starting>^((STATUS \||ERROR \||INFO \|)( jvm 1 \|| wrapper \|)( main \|)))(?P<timestamp> \d{4}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2}).\d{3} \|).*?ERROR(?P<thread>(\s\[[A-Za-z0-9-:\.\s\[]+\])).*(?P<job>(\[[a-zA-Z_\.]+\]|\([a-zA-Z_\.\-]+\)))(?P<message>\s(.*))'],
['exceptionLine','((?P<starting>^((STATUS \||ERROR \||INFO \|)( jvm 1 \|| wrapper \|)( main \|)))(?P<timestamp> \d{4}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2}).\d{3} \|).*(?P<exception>\s([a-zA-Z\.]*Exception))(\s|:)(?P<message>(.*))'],
['causedby','(?P<causedby>(\sCaused by:.+))'],
['causedbyLine','((?P<starting>^((STATUS \||ERROR \||INFO \|)( jvm 1 \|| wrapper \|)( main \|)))(?P<timestamp> \d{4}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2}).\d{3} \|).*?(Caused\sby:)(?P<causedby>(\s.*Exception))(\s|:)(?P<message>(.*))'],
]jdbc logs
[[logs]]
logType = "jdbc"
templateFile = "jdbc.tpl"
reportFileName = "jdbc-report.html"
separator = "|"
splitSeparator = "\n"
lineMatchRegex = '\|(\d{1,10}) ms\|'
fieldsMatchRegex = [
['threadId','^(\d{1,10})\|'],
['dataSourceId','^*(master)'],
['dateAndTime','^*((\d{2})(\d{2})(\d{2})*-([0-1]\d|2[0-3]):([0-5]\d):([0-5]\d):(\d{3}))'],
['executionTime','\|(\d{3,10}) ms\|'],
['category','^*(statement)'],
['statement','(SELECT|INSERT INTO|UPDATE|WITH|DELETE).*\|'],
['sql','\|(INSERT INTO|UPDATE|SELECT|WITH|DELETE)[^?]*$'],
['trace','\/\*(.*?)END'],
]2.- Define the structs that represent the data.
report.go
type FindIssue func(wg *sync.WaitGroup, line string, data *Dataset, rpt *Report) (kv *KeyValue)
type Report struct {
sync.RWMutex
ReportType string
NumberOfFiles int
Issues map[string]interface{}
Template
FindIssue
}
type KeyValue struct {
Key string
Value interface{}
}
type JDBCIssue struct {
Statement string
Occurrences int
TotalTime int
HigherTimeMillis int
AverageTimeMillis int
IssueFileNames map[string]string
Sqls map[string]SqlIssue
}
type SqlIssue struct {
Sql string
Trace []string
}3.- Implement a type function called FindIssue
jdbc.go
func FindIssueJdbs(wg *sync.WaitGroup, line string, d *Dataset, rpt *Report) (kv *KeyValue) {
lf := d.LogFile
mapRegEx := d.MapRegEx
isAndIssue := isJDBCIssue(mapRegEx, line, lf.Separator)
if !isAndIssue {
return nil
}
mapStrValues := make(map[string]string)
for k, v := range mapRegEx {
if matchedField := tools.MatchStringInLine(line, v, lf.Separator); matchedField != "" {
mapStrValues[k] = matchedField
}
}
kv = addJDBCIssue(mapStrValues, lf, rpt)
return kv
}4.- Register that function
func registerReportFunctions(rpt *Report) (error) {
errMsg := fmt.Sprintf("there is not 'FindIssue()' function for this report type %s", rpt.ReportType)
switch rpt.ReportType {
case "jdbc":
rpt.FindIssue = FindIssueJdbs
case "console":
rpt.FindIssue = FindIssueConsole
case "thread-dump":
return errors.New(errMsg)
default:
return errors.New(errMsg)
}
return nil
}5.- Create a text template or html template. Exmples are jdbc.tpl and console.tpl
// here we have an interface
type Publisher interface {
Publish()
}
// No classes but structs
type Template struct {
TemplateFile string
ReportFileName string
}
type Report struct {
sync.RWMutex
ReportType string
NumberOfFiles int
Issues map[string]interface{}
Template //Composition
FindIssue //type function
}
//This is a method
func (rpt *Report) Publish() {
// bunch of code