Log Matching on new data

[Kush-shuk]

Hi, I've been trying to use drain for preprocessing logs for deeplog model

The matching function in Template miner class works perfectly for data it has been trained on, but for some reason it always returns Nonetype when trying to match unseen log data even if some of the logs are identical to the one's in training data.

For example for the following log in training data:-

'WMPLTFMLOG523523\t1676462824978\t2023-02-15 12:07:04.978\t11.16.135.252\t-\t-\t-\t-\tprod\treceiving-api\tunknown\tPROD\t0.0.3118\t4f5f5b1a-205-18654f89e12000\tINFO\tINFO\t-\t-\t-\t-\tapplog.cls=com.expat.move.nim.secure.core.advice.AOPLogger,applog.mthd=beforeControllerMethod,applog.line=34,applog.msg=Entering into [methodName=getHeartBeat] with [requests=[]]\t[]\t[]\t[-]\t[]\t[http-apr-8080-exec-14]\n'

I get the matching cluster_id

but for the exact log in unseen data:-

'WMPLTFMLOG523523\t1681105826004\t2023-04-10 05:50:26.004\t11.16.146.8\t-\t-\t-\t-\tprod\treceiving-api\tunknown\tPROD\t0.0.3393\tffffffffd51aa0d2-192-18769b730d4000\tINFO\tINFO\t-\t-\t-\t-\tapplog.cls=com.expat.move.nim.secure.core.advice.AOPLogger,applog.mthd=beforeControllerMethod,applog.line=34,applog.msg=Entering into [methodName=getHeartBeat] with [requests=[]]\t[]\t[]\t[-]\t[]\t[http-apr-8080-exec-6]\n'

I get a None type return

I passed some extra delimiters as part of the config file when training, could that be causing the issue?

[Superskyyy]

Hi,

I will take a look and try reproduce the abnormal result this weekend.

[Superskyyy]

Just to get more info, are you using the original similarity or jaccard? Please also provide the set of delimiters you used in config file. @Kush-shuk

[milosz7]

I think I had a simillar issue. When adding log message to drain using add_log_message() method drain runs str.strip() method before clasifying the log to a cluster, however when you run the match() method it does not strip the log str passed to the method. I see that your log is ending with a newline character which probably is the issue. Is this desired behavior?