What is the best way to filter logs with lowest cpu consumption?

[alpiua]

Hello,

Can someone help here?

I have a question about "filter" transform.
I have a config like this:

[sources.in]
  type = "file"
  include = ["/opt/app/*/logs/*/*.log"]
  fingerprint.strategy = "device_and_inode"

  [sources.in.multiline]
     start_pattern = '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
     mode = "halt_before"
     condition_pattern = '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
     timeout_ms = 999999999


[transforms.in_filter]
  type = "filter"
  inputs = ["in"]
  condition.type = "check_fields"
  condition."message.not_contains" = "NOT_FOUND - no exchange"
  condition."message.not_regex" = " (INFO|DEBUG|TRACE) "

the problem is that sometimes in the app I have a vast amount of messages coming to files, about 1-2 thousands per one second.
This activity lasts for about one hour.
And vector start consuming 3+ Gb of ram and making performance impact on the running node.

Is there is a better way to reduce ram and cpu consumption while have all messages with "INFO" filtered?

[jszwedko]

Hi @alpi-ua !

That is a lot of memory use! Would you be able to share a few more details:

• The number of files that are typically being processed
• Is the spike distributed across those files? Or is the spike just in one file?
• The rest of the config, including sinks

cc/ @blt @lukesteensen you might be interested in this.

[alpiua]

Here is my config

[api]                                                                                                                                                  
  enabled = true                                                                                                                                       
                                                                                                                                                       
[sources.journal]                                                                                                                                      
  type = "journald"                                                                                                                                    
  exclude_units = ["kernel","vector","telegraf","crond","init.scope","dnf-makecache.service","rsyslog"]                                      
                                                                                                                                                       
[sources.app_stderror]                                                                                                                               
  type = "file"                                                                                                                                        
  include = ["/opt/app/*/logs/stderror.log"]                                                                                                    
  fingerprint.strategy = "device_and_inode"                                                                                                            
  [sources.app_stderror.multiline]                                                                                                                   
    start_pattern = '^[0-9]{4}-[0-9]{2}-[0-9]{2}'                                                                                                      
     mode = "halt_before"                                                                                                                              
     condition_pattern = '^[0-9]{4}-[0-9]{2}-[0-9]{2}'                                                                                                 
     timeout_ms = 999999999                                                                                                                            
                                                                                                                                                       
[sources.app]                                                                                                                                        
  type = "file"                                                                                                                                        
  include = ["/opt/app/*/logs/*/*.log"]                                                                                                          
  fingerprint.strategy = "device_and_inode"                                                                                                            
                                                                                                                                                       
  [sources.app.multiline]                                                                                                                            
     start_pattern = '^[0-9]{4}-[0-9]{2}-[0-9]{2}'                                                                                                     
     mode = "halt_before"                                                                                                                              
     condition_pattern = '^[0-9]{4}-[0-9]{2}-[0-9]{2}'                                                                                                 
     timeout_ms = 999999999                                                                                                                            
                                                                                                                                                                                                                                                     
[transforms.app_filter]                                                                                                                              
  type = "filter"                                                                                                                                      
  inputs = ["app"]                                                                                                                                   
  condition.type = "check_fields"                                                                                                                      
  condition."message.not_contains" = "NOT_FOUND - no exchange"                                                                                         
  condition."message.not_regex" = " (INFO|DEBUG|TRACE) "                                                                                               
                                                                                                                                                       
                                                                                                                                                       
[sinks.app_prod]                                                                                                                                     
  type = "vector"                                                                                                                                      
  inputs = ["app_filter","app_stderror"]                                                                                                           
  address = "server:5001"                                                                                                               
  healthcheck = true                                                                                                                                   
  buffer.max_size = 10100000                                                                                                                           
  buffer.type = "disk"                                                                                                                                 
  buffer.when_full = "block" 

[sinks.system]                                                                                                                                         
  type = "vector"                                                                                                                                      
  inputs = ["journal"]                                                                                                                                 
  address = "server:5000"                                                                                                               
  healthcheck = true                                                                                                                                   
  buffer.max_size = 10100000                                                                                                                           
  buffer.type = "disk"                                                                                                                                 
  buffer.when_full = "block"

btw, with 'vector top' I can't see any activity on transform.app_filter - it showing N/A nevertheless showing activity on source.app.
I have timeout_ms 99999999 because of bug I faced before

[blt]

@alpi-ua can you share which version of vector your using with us? We did fix a pretty serious memory leak in #7014 and it'd be good to know if you're using a version that still has this issue, which should be the case unless you're using nightly builds -- @jszwedko correct me if I'm wrong -- until our next release is cut.

[alpiua]

As described in #7014 - is it fixing memory leaks related only with metrics scenarios, or everything beside as well ?

[blt]

#7014 fixes a general memory leak, one that's expressed whenever vector's internal metrics and tracing mechanisms are turned on, which they are by default.

[alpiua]

Ow, so I might want to turn them of completely if I want to reduce the impact on the node?

[blt]

Probably not? There are some undocumented environment variables that will disable bits and pieces of these mechanisms but not their integration into vector and doing so will limit the self-reporting that vector is able to do. When I was working on #7014 I couldn't find that enabling these environment variables, when set up, had any impact on throughput compared to vector default.

[alpiua]

I still have issues even with a new version.
It might be related to buffers. Created ticket #7246

[alpiua]

Want to clarify my question - is regexp filter fast enough in my case with such amount of logs, or I better try to use remap instead?

[dojci]

Hey @alpi-ua!

What do you mean by 3+ Gb of ram, what part of memory, RSS, VSZ, cache ?

[alpiua]

Unfortunately, I forgot to pay attention on this when it happened.
It was like more than 30% of total consuming on the node with 8Gb ram installed (by top %MEM).
So I suggest a 3Gb is fair enough comparing to total ram. Agree that that is not very fair.
Normally vector takes no more than 5-7% with SHR about 47MiB and RES 180MiB, with 1-2 instances running.

[alpiua]

Well, I was able to catch the moment with high consumption

Actually I wonder why consumption is so high, because vector was only processing relatively small amount of data - was filtering messages from a file

And it was a new version of vector.

[JeanMertz]

The issue tracking this problem (#7246) has been resolved since vector 0.14 We've as well since significantly improved performance generally for Vector. The latest version of vector is 0.23.3.

I'll mark this discussion as resolved, but you are welcome to comment here if that turns out to be false.