diff --git a/Content/Guides/syslog-ng-guide-admin/configuring-destinations-mongodb.htm b/Content/Guides/syslog-ng-guide-admin/configuring-destinations-mongodb.htm index d0466bedc..5141eef46 100644 --- a/Content/Guides/syslog-ng-guide-admin/configuring-destinations-mongodb.htm +++ b/Content/Guides/syslog-ng-guide-admin/configuring-destinations-mongodb.htm @@ -1,20 +1,21 @@  - -

mongodb(): Storing messages in a MongoDB database

+

mongodb: Storing messages in a MongoDB database

-

The mongodb() driver sends messages to a MongoDB database. MongoDB is a schema-free, document-oriented database. For the list of available optional parameters, see mongodb() destination options.

+

The mongodb() driver sends messages to a MongoDB database. MongoDB is a schema-free, document-oriented database. For the list of available optional parameters, see mongodb() destination options.

+ +
Declaration
mongodb(parameters);

The mongodb() driver does not support creating indexes, as that can be a very complex operation in MongoDB. If needed, the administrator of the MongoDB database must ensure that indexes are created on the collections.

-

The mongodb() driver does not add the _id field to the message: the MongoDB server will do that automatically, if none is present. If you want to override this field from , use the key() parameter of the value-pairs() option.

+

The mongodb() driver does not add the _id field to the message: the MongoDB server will do that automatically, if none is present. If you want to override this field from , use the key() parameter of the value-pairs() option.

The mongodb() driver is compatible with MongoDB server version 1.4 and newer.

- +
Example: Using the mongodb() driver
@@ -30,7 +31,7 @@
-

The following example shows the same setup using the deprecated libmongo-client syntax (as used in version 3.7), and is equivalent with the previous example.

destination d_mongodb {
+            
The following example shows the same setup using the deprecated libmongo-client syntax, and is equivalent with the previous example.
destination d_mongodb {
     mongodb(
         servers("localhost:27017")
         database("syslog")
diff --git a/Content/Guides/syslog-ng-guide-admin/date-parser-options.htm b/Content/Guides/syslog-ng-guide-admin/date-parser-options.htm
index 0b7649e99..21ec5246a 100644
--- a/Content/Guides/syslog-ng-guide-admin/date-parser-options.htm
+++ b/Content/Guides/syslog-ng-guide-admin/date-parser-options.htm
@@ -1,9 +1,7 @@
 
 
     
-        
         
-        
     
     
         
Options of date-parser() parsers

@@ -13,26 +11,59 @@ 
Options of date-parser() parsers

         
         
The date-parser() parser has the following options.

         

-            
format()

-            
+            
flags()

+            
+            
+            
             
             
-                        
-                        
+                        
+                        
-                        
-                        
+                        
+                        

                 
                     
Synopsis:format(string)Type:
+
+
+						guess-timezone
                     

                     
Default:
-                            
 

-                        		Default:
+
+
+						empty string
                     

                 
                 
                 
             

-            
Description: Specifies the format how  should parse the date. You can use the following format elements:
%%      PERCENT
+            
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp. For example:
date-parser(flags(guess-timezone));

+        

+		

+        
format()

+        
+        
+        
+            
+                
+                    
+                    
+                
+                
+                    
+                    
+                
+            
+            +            
+        
Synopsis:
+									
+
+                            format(string)
Default:
+									
+
+                            
+                    



+        
Description: Specifies the format how  should parse the date. You can use the following format elements:
%%      PERCENT
 %a      day of the week, abbreviated
 %A      day of the week
 %b      month abbr
@@ -71,118 +102,94 @@ 
format()

 %X      time format: 21:05:57
 %y      year (2 digits)
 %Y      year (4 digits)
-%Z      timezone in ascii format (for example, PST), or in format -/+0000
-%z      timezone in ascii format (for example, PST), or in format -/+0000  (Required element)

-            
-                -                -                
-                    
-                        
-                        
-                    
-                
-            




-                            

-                                
-                            

-                        		
-                            
CAUTION: When using the %z and %Z format elements, consider that while %z strictly expects a specified timezone, and triggers a warning if the timezone is not specified, %Z does not trigger a warning if the timezone is not specified.

-                            
For further information about the %z and %Z format elements, see the 'DESCRIPTION' section on the srtptime(3) - NetBSD Manual Pages.

-                        

-            
For example, for the date 01/Jan/2016:13:05:05 PST use the following format string: format("%d/%b/%Y:%H:%M:%S %Z")

-            
-            
-        

-        
+%Z      timezone in ascii. eg: PST
+%z      timezone in format -/+0000

+        
For example, for the date 01/Jan/2016:13:05:05 PST use the following format string: format("%d/%b/%Y:%H:%M:%S %Z")

+        
         
-        

-            
flags()

-            
-            
-            
-            
-            
-                
-                    
-                        
+                
+            
+            +            
+        
Type:
-
+		
+		

+        
time-stamp()

+        
+        
+        
+            
+                
+                    
-                        
-                    
-                    
-                        
+                
+                
+                    
-                        
-                    
-                
-                -                
-            
Synopsis:
+									
 
                             guess-timezone
Default:
-
+                    stamp | recvd
Default:
+									
 
                             empty string



-            
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp. For example:
date-parser(flags(guess-timezone));

-        

-        

-            
time-stamp()

-            
-            
-            
-            
-            
-                
-                    
-                        
+                
+            
+            +            
+        
Synopsis:
-
+                    stamp



+        
Description: Determines if the parsed date values are treated as sent or received date. If you use time-stamp(),  adds the parsed date to the S_ macros (corresponding to the sent date). If you use time-zone(recvd),  adds the parsed date to the R_ macros (corresponding to the received date).

+		

+		

+        
time-zone()

+        
+        
+        
+            
+                
+                    
-                        
-                    
-                    
-                        
+                
+                
+                    
-                        
-                    
-                
-                -                
-            
Synopsis:
+									
 
                             stamp | recvd
Default:
-
+                    timezone(string)
Default:
+									
 
                             stamp



-            
Description: Determines if the parsed date values are treated as sent or received date. If you use time-stamp(stamp),  adds the parsed date to the S_ macros (corresponding to the sent date). If you use time-stamp(recvd),  adds the parsed date to the R_ macros (corresponding to the received date).

-        

-        

-            
time-zone()

-            
-            
-            
-            
-            
-                
-                    
-                        
+                
+            
+            +            
+        
Synopsis:
-
+                    
+                    



+        
Description: If this option is set,  assumes that the parsed timestamp refers to the specified timezone. The timezone set in the time-zone() option overrides any timezone information parsed from the timestamp.

+        
+        
+		

+		

+        
value()

+        
+        
+        
+            
+                
+                    
-                        
-                    
-                    
-                        
+                
+                
+                    
-                        
-                    
-                
-                -                
-            
Synopsis:
+									
 
                             time-zone(string)
Default:
-
+                    string
Default:
+									
 
                             
-                            
 

-                        



-            
Description: If this option is set,  assumes that the parsed timestamp refers to the specified timezone. The timezone set in the time-zone() option overrides any timezone information parsed from the timestamp.

-            
-            
-        

+                    		
+                    



+		
This option is only available in  4.0 and later versions.

+        
Description: This option instructs the date-parser() to store the timestamp in a name-value pair specified in value(), instead of editing the timestamp value of the log message.

+		

     
 
\ No newline at end of file
diff --git a/Content/Guides/syslog-ng-guide-admin/filters-comparing.htm b/Content/Guides/syslog-ng-guide-admin/filters-comparing.htm
index 0a05b7315..3cfa4866c 100644
--- a/Content/Guides/syslog-ng-guide-admin/filters-comparing.htm
+++ b/Content/Guides/syslog-ng-guide-admin/filters-comparing.htm
@@ -1,7 +1,6 @@
 
 
     
-        
     
     
         
Comparing macro values in filters

@@ -27,13 +26,37 @@ 
Comparing macro values in filters

         
         
         
-        
Starting with  version 3.24 F1, it is also possible to compare macro values and templates as numerical and string values. String comparison is alphabetical: it determines if a string is alphabetically greater or equal to another string. Use the following syntax to compare macro values or templates. For details on macros and templates, see Customize message format using macros and templates.
filter <filter-id>
+        
Starting with  version 3.24 F1, it is also possible to compare macro values and templates as numerical and string values. String comparison is alphabetical: it determines if a string is alphabetically greater or equal to another string. Use the following syntax to compare macro values or templates. For details on macros and templates, see Customizing message format using macros and templates.
filter <filter-id>
         {"<macro-or-template>" operator "<value-or-macro-or-template>"};

+	
String and numerical comparison

+	
 versions prior to 4.0 used separate operators for string comparisons (for example, eq). In version 4.0 and onwards, mathematical symbols can be used as operators (==, !=, >=), and  automatically determines how to compare the arguments based on their type.

+	
    
+		
  • If both sides of the comparison are strings, the comparison is string.
    • 
+		
  • If at least one of the argument is numeric, the comparison is numeric.
    • 
+		
  • Numbers not enclosed by quotation marks, are recognized as numeric.
    • 
+		
  • It is possible to explicitly type-cast an argument as numeric.
    • 
+	

+	
Examples:

+	
    
+		
  • if ("${.apache.httpversion}" == 1.0)
    
+		
    The right side of the argument is 1.0 (a floating point literal), so the comparison is numeric.
    
+		
    • 
+		
  • if (double("${.apache.httpversion}") == "1.0")
    
+		
    The left side of the argument is explicitly type cast into double, the right side is a string (note the use of quotation marks), so the comparison is numeric.
    
+		
    • 
+		
  • if ("${.apache.request}" == "/wp-admin/login.php")
    
+		
    The left side of the argument is not type-cast, so it remains a string. The right side of the argument is also a string, so the comparison is string.
    
+		
    • 
+	

+

+	
NOTE: The old string operators can still be used for the sake of backwards compatibility, but it is advised and easier to use the numeric operators for strings as well.

+ 

+ 

         

             
Example: Comparing macro values in filters

             
The following expression selects log messages containing a PID (that is, ${PID} macro is not empty):
filter f_pid {"${PID}" !=""};

             
The following expression selects log messages that do not contain a PID. Also, it uses a template as the left argument of the operator and compares the values as strings:
filter f_pid {"${HOST}${PID}" eq "${HOST}"};

-            
The following example selects messages with priority level higher than 5.
filter f_level {"${LEVEL_NUM}" > "5"};

+            
The following example selects messages with priority level 4 or higher.
filter f_level {"${LEVEL_NUM}" > "5"};

         

         
Note that:

         
    
@@ -44,7 +67,7 @@ 
    Example: Comparin
                 
    The $ character must be used before macros.
    
             
             
  • 
-                
    Using comparator operators can be equivalent to using filter functions, but is somewhat slower. For example, using "${HOST}" eq "myhost" is equivalent to using host("myhost" type(string)).
    
+                
    Using comparator operators can be equivalent to using filter functions, but is somewhat slower. For example, using "${HOST}" eq "myhost" is equivalent to using host("myhost" type(string)).
    
             
    • 
             
  • 
                 
    You can use any macro in the expression, including user-defined macros from parsers and results of pattern database classifications.
    
@@ -57,7 +80,7 @@ 
    Example: Comparin
             
    • 
         

         
The following operators are available:

-        
+        
         
     
 
\ No newline at end of file
diff --git a/Content/Guides/syslog-ng-guide-admin/json-parser.htm b/Content/Guides/syslog-ng-guide-admin/json-parser.htm
index 8694a8ee5..2b9569282 100644
--- a/Content/Guides/syslog-ng-guide-admin/json-parser.htm
+++ b/Content/Guides/syslog-ng-guide-admin/json-parser.htm
@@ -2,7 +2,6 @@
 
     
         
-        
     
     
         
JSON parser

@@ -15,7 +14,7 @@ 
JSON parser

         
         
         
JavaScript Object Notation (JSON) is a text-based open standard designed for human-readable data interchange. It is used primarily to transmit data between a server and web application, serving as an alternative to XML. It is described in RFC 4627. The  application can separate parts of incoming JSON-encoded log messages to name-value pairs. For details on using value-pairs in  see Structuring macros, metadata, and other value-pairs.

-        
You can refer to the separated parts of the JSON message using the key of the JSON object as a macro. For example, if the JSON contains {"KEY1":"value1","KEY2":"value2"}, you can refer to the values as ${KEY1} and ${KEY2}. If the JSON content is structured,  converts it to dot-notation-format. For example, to access the value of the following structure {"KEY1": {"KEY2": "VALUE"}}, use the ${KEY1.KEY2} macro.

+        
You can refer to the separated parts of the JSON message using the key of the JSON object as a macro. For example, if the JSON contains {"KEY1":"value1","KEY2":"value2"}, you can refer to the values as ${KEY1} and ${KEY2}. If the JSON content is structured,  converts it to dot-notation-format. For example, to access the value of the following structure {"KEY1": {"KEY2": "VALUE"}}, use the ${KEY1.KEY2} macro.

         @@ -29,21 +28,21 @@ 
JSON parser

             
             

                         
                     
                     
Caution: 
-                        
If the names of keys in the JSON content are the same as the names of  soft macros, the value from the JSON content will overwrite the value of the macro. For example, the {"PROGRAM":"value1","MESSAGE":"value2"} JSON content will overwrite the ${PROGRAM} and ${MESSAGE} macros. To avoid overwriting such macros, use the prefix() option.

-                        
Hard macros cannot be modified, so they will not be overwritten. For details on the macro types, see Hard vs. soft macros.

+                        
If the names of keys in the JSON content are the same as the names of  soft macros, the value from the JSON content will overwrite the value of the macro. For example, the {"PROGRAM":"value1","MESSAGE":"value2"} JSON content will overwrite the ${PROGRAM} and ${MESSAGE} macros. To avoid overwriting such macros, use the prefix() option.

+                        
Hard macros cannot be modified, so they will not be overwritten. For details on the macro types, see Hard vs. soft macros.

                     		
                 

             
         

         

             
NOTE: 
-
-			The JSON parser currently supports only integer, double and string values when interpreting JSON structures. As syslog-ng does not handle different data types internally, the JSON parser converts all JSON data to string values. In case of boolean types, the value is converted to 'TRUE' or 'FALSE' as their string representation.

-            
The JSON parser discards messages if it cannot parse them as JSON messages, so it acts as a JSON-filter as well.

+			When using the json-parser(),  converts all elements of the JSON object to name-value pairs. Any type information carried by the incoming JSON object is preserved, and automatically propagated to other  components (for example, a destination) if the component supports types. Elements without a type are handled as strings. JSON lists (arrays) are converted to lists, and can be manipulated using the $(list-*) template functions.

+            
Prior to version 4.0,  handled all data as string.

+			
The JSON parser discards messages if they cannot be parsed as JSON messages, therefore acting as a JSON-filter as well.

         

-        
To create a JSON parser, define a parser that has the json-parser() option. Defining the prefix and the marker are optional. By default, the parser will process the ${MESSAGE} part of the log message. To process other parts of a log message with the JSON parser, use the template() option. You can also define the parser inline in the log path.

+        
To create a JSON parser, define a parser that has the json-parser() option. Defining the prefix and the marker are optional. By default, the parser will process the ${MESSAGE} part of the log message. To process other parts of a log message with the JSON parser, use the template() option. You can also define the parser inline in the log path.

         

-            
Declaration:
parser parser_name {
+            
Declaration
parser parser_name {
     json-parser(
         marker()
         prefix()
@@ -52,18 +51,13 @@ 
Declaration:
parser parser_name {
         

         

             
Example: Using a JSON parser

-            
In the following example, the source is a JSON encoded log message. The syslog parser is disabled, so that  does not parse the message: flags(no-parse). The json-parser inserts ".json." prefix before all extracted name-value pairs. The destination is a file that uses the format-json template function. Every name-value pair that begins with a dot (".") character will be written to the file (dot-nv-pairs). The log line connects the source, the destination and the parser.
source s_json {
-    network(
-        port(21514
-        flags(no-parse)
-    );
+            
In the following example, the source is a JSON encoded log message. The syslog parser is disabled, so that  does not parse the message: flags(no-parse). The json-parser inserts ".json." prefix before all extracted name-value pairs. The destination is a file, that uses the format-json template function. Every name-value pair that begins with a dot (".") character will be written to the file (dot-nv-pairs). The log line connects the source, the destination and the parser.
source s_json {
+    network(port(21514) flags(no-parse));
 };
 
 destination d_json {
-    file(
-        "/tmp/test.json"
-        template("$(format-json --scope dot-nv-pairs)\n")
-    );
+    file("/tmp/test.json"
+        template("$(format-json --scope dot-nv-pairs)\n"));
 };
 
 parser p_json {
@@ -76,17 +70,12 @@ 
Example: Using a JSON parser

     destination(d_json);
 };

             
You can also define the parser inline in the log path.
source s_json {
-    network(
-        port(21514)
-        flags(no-parse)
-    );
+    network(port(21514) flags(no-parse));
 };
 
 destination d_json {
-    file(
-        "/tmp/test.json"
-        template("$(format-json --scope dot-nv-pairs)\n")
-    );
+    file("/tmp/test.json"
+        template("$(format-json --scope dot-nv-pairs)\n"));
 };
 
 log {
diff --git a/Content/Guides/syslog-ng-guide-admin/reference-destination-mongodb.htm b/Content/Guides/syslog-ng-guide-admin/reference-destination-mongodb.htm
index 448b3d0be..eb59a018e 100644
--- a/Content/Guides/syslog-ng-guide-admin/reference-destination-mongodb.htm
+++ b/Content/Guides/syslog-ng-guide-admin/reference-destination-mongodb.htm
@@ -3,14 +3,90 @@
     
         
         
-        
     
     
         
mongodb() destination options

         
         
         
The mongodb() driver sends messages to a MongoDB database. MongoDB is a schema-free, document-oriented database.

+        
+        
         
The mongodb() destination has the following options:

+        

+            
bulk()

+            
+            
+            
+                
+                    
+                        
+                        
+                    
+                    
+                        
+                        
+                    
+                
+                +                
+            
Type: 
+
+                            yes|no
Default: 
+
+                            yes



+            
Description: Toggles bulk insertion, setting to no forces the old behavior (each log is inserted one by one into the MongoDB).

+            

+                
NOTE: Bulk sending is only efficient if the used collection is constant (e.g. not using templates) or the used template does not lead to too many collections switching within a reasonable time range.

+            

+        

+        

+            
bulk_unordered()

+            
+            
+            
+                
+                    
+                        
+                        
+                    
+                    
+                        
+                        
+                    
+                
+                +                
+            
Type: 
+
+                            yes|no
Default: 
+
+                            no



+            
Description: Toggles unordered bulk operations.

+        

+        

+            
bulk_bypass_validation()

+            
+            
+            
+                
+                    
+                        
+                        
+                    
+                    
+                        
+                        
+                    
+                
+                +                
+            
Type: 
+
+                            yes|no
Default: 
+
+                            no



+            
Description: Toggles MongoDB bulk operation validation.

+        

         

             
collection()

             
@@ -18,18 +94,30 @@ 
col
             
-                        
-                        
+                        
+                        
-                        
+                        

                 
                     
Type:templateType: 
+
+                            string
                     

                     
Default:Default: 
+
+                            
                         messages
                     

                 
                 
                 
             

-            
Description: The name of the MongoDB collection where the log messages are stored (collections are similar to SQL tables). You can use templates to change the collection dynamically based on the source or the content of the message, for example, collection("${HOST}").

+            
Description: The name of the MongoDB collection where the log messages are stored (collections are similar to SQL tables). The collection() option supports template functions and macros as well.

+            

+                
Example: using the mongodb() destination with a template embedded in the collection() option

+                
Using the following example configuration, the mongodb() destination sends incoming messages into separate MongoDB collections (for example, localhost_messages and anotherhost_messages) based on the HOST field of the message :
mongodb(
+  uri("mongodb://host/syslog?wtimeoutMS=10000&socketTimeoutMS=10000&connectTimeoutMS=10000&serverSelectionTimeoutMS=5000")
+  collection("${HOST}_messages")
+  workers(8)
+);

+            

             @@ -49,32 +137,22 @@ 
col
                 
             

                 
                 


         

-        
-        
-        
-        
-        
+        
         
-        
+        
         
-        
+        
         
-        
+        
         
-        
-        
-        
-        
-        
+        
         
         

-            
+            
             
             
For MongoDB operations,  uses a one-minute timeout: if an operation times out,  assumes the operation has failed.

         

-        
-        
-        
+        
         
         

             
uri()

@@ -83,45 +161,78 @@ 
uri()

             
-                        
+                        
-                        
+                        

                 
                     
Type:Type: 
+
+                            
                         string
                     

                     
Default:Default: 
+
+                            
                         mongodb://127.0.0.1:27017/syslog?wtimeoutMS=60000&socketTimeoutMS=60000&connectTimeoutMS=60000
                     

                 
                 
                 
             

-            
Description: Available in syslog-ng OSE 3.8 and later. Please refer to the MongoDB URI format documentation for detailed syntax.

+            
Description: Refer to the MongoDB URI format documentation for detailed syntax.

+        

+        

+            
+            
+            
+            
+        

+        

+            

+                
workers()

+                
+                
+                
+                    
+                        
+                            
+                            
+                        
+                        
+                            
+                            
+                        
+                    
+                    +                    
+                
Type:integer
Default:1
+							



+                
+                
+            

         

-        
-        
-        
-        
         

-            
workers()

-            
-            
+            
write_concern()

+            
+            
             
-                        
-                        
+                        
+                        
-                        
-                        
+                        
+                        
-                +                +                

                 
                     
Type:integerAccepted values: 
+
+                            
+                            
unacked|acked|majority|a number greater than 0

+                        		
                     

                     
Default:1
-                        Default: 
+
+                            acked
                     

                 


             

-            

-                
Description: Specifies the number of worker threads (at least 1) that  uses to send messages to the server. Increasing the number of worker threads can drastically improve the performance of the destination.

-                
-            

+            
Description: Sets write concern mode of MongoDB operations, both bulk and single.

         

     
 
\ No newline at end of file
diff --git a/Content/Guides/syslog-ng-guide-admin/reference-template-functions.htm b/Content/Guides/syslog-ng-guide-admin/reference-template-functions.htm
index 6c7db52cc..4d1db4cf9 100644
--- a/Content/Guides/syslog-ng-guide-admin/reference-template-functions.htm
+++ b/Content/Guides/syslog-ng-guide-admin/reference-template-functions.htm
@@ -1,217 +1,53 @@
 
 
     
-        
         
-        
     
     
-        
Template functions of 

+        
Template functions of 

         
The following template functions are available in .

         

             
base64-encode

             
             
-            
Syntax:
$(base64-encode argument)

+            
Syntax:
$(base64-encode argument)

             

-            
Description: You can use the base64-encode template function to base64-encode strings and macros. The template function can receive multiple parameters (maximum 64). In this case,  joins the parameters into a single string and encodes this string. For example, $(base64-encode string1 string2) is equivalent to $(base64-encode string1string2).

+            
Description: You can use the base64-encode template function to base64-encode strings and macros. The template function can receive multiple parameters (maximum 64). In this case,  joins the parameters into a single string and encodes this string. For example, $(base64-encode string1 string2) is equivalent to $(base64-encode string1string2).

             
Available in  version 3.187.0.11 and later.

         

         

             
basename

             
             
-            
Syntax:
$(basename argument)

+            
Syntax:
$(basename argument)

             

-            
Description: Returns the filename from an argument (for example, a macro: $(basename ${FILE_NAME})) that contains a filename with a path. For example, $(basename "/var/log/messages.log") returns messages.log. To extract the path, use the dirname template function.

+            
Description: Returns the filename from an argument (for example, a macro) that contains a filename with a path. For example, $(basename "/var/log/messages.log") returns messages.log. To extract the path, use the dirname template function.

             
Available in  version 3.107.0.3 and later.

         

-        

-            
ceil

-            
-            
-            
Syntax:
$(ceil argument)

-            

-            
Description: Rounds a floating-point number upwards to the nearest integer. For example, $(ceil 1.5) is 2, $(ceil -1.5) is -1. See also the floor and round template functions.

-        

-        

-            
context-lookup

-            
-            
-            
Syntax:
$(context-lookup [option] condition value-to-select)

-            

-            
Description: The context-lookup template function can search a message context when correlating messages (for example, when you use a pattern database or the grouping-by parser). The context-lookup template function requires a condition (a filter or a string), and returns a specific macro or template of the matching messages (for example, the ${MESSAGE}) as a list. It works similarly to the $(grep) template function, but it escapes its output properly, so that the returned value is a list that can be processed with other template functions that work on lists, for example, $(list-slice).

-            

-                
Example: Using the context-lookup template function

-                
The following example selects the message of the context that has a username name-value pair with the root value, and returns the value of the tags name-value pair.
$(context-lookup ("${username}" == "root") ${tags})

-            

-            
To limit the number of matches that the template function returns, use the --max-count option, for example, $(context-lookup --max-count 5 ("${username}" == "root") ${tags}). If you do not want to limit the number of matches, use --max-count 0.

-            
You can to specify multiple name-value pairs as parameters, separated with commas. If multiple messages match the condition of context-lookup, these will be returned also separated by commas. This can be used for example, to collect the email recipients from postfix messages.

-            
Available in  version 3.10 and later.

-        

-        

-            
context-values

-            
-            
-            
Syntax:
$(context-values $name-value1 $name-value2 ...)

-            

-            
Description: The context-values template function returns a list of every occurrence of the specified name-value pairs from the entire context. For example, if the context contains multiple messages, the $(context-values ${HOST}) template function will return a comma-separated list of the ${HOST} values that appear in the context.

-            
Available in  version 3.10 and later.

-        

         

             
dirname

             
             
-            
Syntax:
$(dirname argument)

+            
Syntax:
$(dirname argument)

             

-            
Description: Returns the path (without the filename) from an argument (for example, a macro: $(basename ${FILE_NAME}) that contains a filename with a path. For example, $(dirname "/var/log/messages.log") returns /var/log path. To extract the filename, use the basename template function.

+            
Description: Returns the path (without the filename) from an argument (for example, a macro) that contains a filename with a path. For example, $(dirname "/var/log/messages.log") returns /var/log path. To extract the filename, use the basename template function.

             
Available in  version 3.107.0.3 and later.

         

         

             
echo

             
             
-            
Syntax:
$(echo argument)

+            
Syntax:
$(echo argument)

             

-            
Description: Returns the value of its argument. Using $(echo ${HOST}) is equivalent to ${HOST}.

+            
Description: Returns the value of its argument. Using $(echo ${HOST}) is equivalent to ${HOST}.

         

         

             
env

             
             
-            
Syntax:
$(env <environment-variable>)

+            
Syntax:
$(env <environment-variable>)

             

-            
Description: Returns the value of the specified environment variable. Available in 3.57 and later.

-        

-        

-            
explode

-            
-            
-            
Syntax:
$(explode <separator> <string1> <string2> ...)

-            

-            
Description: Turns a string separated by a specific character into a list.  You can also use the   implode template function, which turns a list into a string combining the pieces together with a separator. Available in 3.217 and later.

-            

-                
Example: Using the explode template function

-                
The following configuration example turns strings into a list. If there are several strings,  looks for a separator within each individual string. For example, string 2 is separated as string, 2 in the example below:

-                
-                    -                    -                    
-                        
-                            
-                            
-                        
-                        
-                            
-                            
-                        
-                    
-                



ConfigurationResult
$(explode ';' string1;string 2;string3;string4)"string1,string,2,string3,string4"

-                
Enclose the strings in double-quotes or apostrophes and string 2 is separated as shown below:

-                
-                    -                    -                    
-                        
-                            
-                            
-                        
-                        
-                            
-                            
-                        
-                    
-                



ConfigurationResult
$(explode ' ' 'string1 string 2 string3 string4 string5')"string1,string 2,string3,string4,string5"

-                
The following examples replace the separator ';' character with a ',' character:

-                
-                    -                    -                    
-                        
-                            
-                            
-                        
-                        
-                            
-                            
-                        
-                        
-                            
-                            
-                        
-                    
-                



ConfigurationResult
$(implode ',' $(explode ';' 'string1;string2;string3'))
-                            "string1,string2,string3"

-                                
$(explode ';' 'string1;string2;string3;string4;string5')
-                                

-                            		
-                                
"string1,string2,string3,string4,string5"

-                            

-            

-        

-        

-            
filter

-            
-            
Syntax:
$(filter <filter-expression> <list>)

-            

-            
Description: Runs the filter expression on each element of a given list, and returns only those list elements that meet the requirements of the filter expression. The current value is referred by $_, similarly to the map template function.

-            

-                
NOTE: You can use macros, logical expressions, and template functions inside the expression.

-            

-            
Available in  version 3.30 and later.

-            

-                
Example: using the filter template function in your configuration

-                
When used in configuration as seen in the example, the filter template function filters even numbers from an input list of 0, 1, 2 and 3:
log {
-  source { example-msg-generator(num(1) values(INPUT => "0,1,2,3")); };
-  destination {
-     file("/dev/stdout"
-           template("$(filter ('$(% $_ 2)' eq '0') $INPUT)\n)")
-     );
-  };
-};
-

-                
The returned values are 0 and 2.

-            

-            
Parameters:
-            

-            
    
-                
  • 
-                    
    <filter-expression>
-                    
    
-                    
    Mandatory parameter.
    
-                    
    The <filter-expression> parameter can be:
    
-                    
      
-                        
    • 
-                            
      a comparison
      
-                        
      • 
-                        
    • 
-                            
      a filter
      
-                        
      • 
-                        
    • 
-                            
      a logical expression built from filters (using and, or, and not)
      
-                        
      • 
-                    
    
-                    
    
-                        
    NOTE: When using the <filter-expression> parameter, you can refer other template functions, or use macros.
    
-                    
    
-                    
    
-                        
    NOTE: To refer to the variable bound to the current element of the list, use $_ macro.
    
-                    
    
-                    
    
-                        
    Examples for <filter-expression>
    
-                        
    The following examples illustrate several ways that you can use a single filter, or a logical expression built from several filters.
    ('1' == '1')
-('$_' le '1')
-('$(% $_ 2)' eq '0')
-('$_' le '1') and ('$(% $_ 2)' eq '0')
-
    
-                    
    
-                
    • 
-                
  • 
-                    
    list
-                    
    
-                    
    Mandatory parameter. A  list.
    
-                
    • 
-            

+            
Description: Returns the value of the specified environment variable. Available in 3.57 and later.

         

         

             
format-cef-extension

@@ -221,7 +57,7 @@ 

             
-            
 version 5 F63.8 includes a new template function (format-cef-extension) to format name-value pairs as ArcSight Common Event Format extensions. Note that the template function only formats the selected name-value pairs, it does not provide any mapping. There is no special support for creating the prefix part of a Common Event Format (CEF) message. Note that the order of the elements is random. For details on the CEF extension escaping rules format, see the ArcSight Common Event Format.

+            
 includes a template function (format-cef-extension) to format name-value pairs as ArcSight Common Event Format extensions. Note that the template function only formats the selected name-value pairs, it does not provide any mapping. There is no special support for creating the prefix part of a Common Event Format (CEF) message. Note that the order of the elements is random. For details on the CEF extension escaping rules format, see the ArcSight Common Event Format.

             
You can use the value-pairs that  stores about the log message as CEF fields. Using value-pairs, you can:

             

                 
The following example selects every value-pair that has a name beginning with .cef., but removes the .cef. prefix from the key names.
template("$(format-cef-extension --subkeys .cef.)\n")

                 
The following example shows how to use this template function to store log messages in CEF format:
destination d_cef_extension {
-    file("/var/log/messages.cef" template("${ISODATE} ${HOST} $(format-cef-extension --scope selected_macros --scope nv_pairs)\n"));
+  file("/var/log/messages.cef" template("${ISODATE} ${HOST} $(format-cef-extension --scope selected_macros --scope nv_pairs)\n"));
 };
@@ -261,18 +100,14 @@
Example: Using the format-cef-extension template function
format-cim
-

Syntax:

$(format-cim)
+

Syntax:

$(format-cim)

-

Description: Formats the message into Splunk Common Information Model (CIM) format. Applications that can receive messages in CIM format include Kibana, logstash, and Splunk. Applications that can be configured to log into CIM format include nflog and the Suricata IDS engine.

destination d_cim {
-    network(
-        "192.168.1.1"
-        template("$(format-cim)\n")
-    );
+            
Description: Formats the message into Splunk Common Information Model (CIM) format. Applications that can receive messages in CIM format include Kibana, logstash, and Splunk. Applications that can be configured to log into CIM format include nflog and the Suricata IDS engine.
destination d_cim {
+    network("192.168.1.1" template("$(format-cim)\n"));
 };

             
You can find the exact source of this template function in the  GitHub repository.

             

-                
NOTE: 
-				To use the format-cim() template function,  must be compiled with JSON support. For details, see Compiling options of [%=General.OSE%]. To see if your  binary was compiled with JSON support, execute the syslog-ng --version command.

+                
NOTE: To use the format-cim() template function,  must be compiled with JSON support. To see if your  binary was compiled with JSON support, execute the syslog-ng --version command.

             

         
         

@@ -283,9 +118,9 @@ 

             
             
-            
Syntax:
$(format-ewmm)

+            
Syntax:
$(format-ewmm)

             

-            
Description: The format-ewmm template function converts the message into the Enterprise-wide message model (EWMM) format. Available in version 7.0.93.16 and later.

+            
Description: The format-ewmm template function converts the message into the Enterprise-wide message model (EWMM) format. Available in version 7.0.93.16 and later.

             
             
         

@@ -305,49 +140,29 @@ 
Syntax:
$(format-flat-json parameters)

             

-            
Description: The format-flat-json template function is identical to the format-json template function, but nested JSON objects are flattened in the output. If you have to forward your log messages in JSON format, but the receiving application cannot handle nested JSON objects, use the format-flat-json template function.

+            
Description: The format-flat-json template function is identical to the format-json template function, but nested JSON objects are flattened in the output. If you have to forward your log messages in JSON format, but the receiving application cannot handle nested JSON objects, use the format-flat-json template function.

             

                 
Example: Flattened JSON output

                 
The following example shows the difference between nested and flattened JSON objects.

                 
    
                     
  • 
                         
    The output of $(format-json a.b.c=1) is a nested JSON object (whitespace added for better readability):
    {
-    "a": {
-        "b": {
-        "c": "1"
-        }
-    }
+   "a": {
+     "b": {
+      "c": "1"
+     }
+   }
 }
    
                     
    • 
                     
  • 
                         
    The output of $(format-flat-json a.b.c=1) is a flattened JSON object (whitespace added for better readability):
    {
-    "a.b.c": "1"
+"a.b.c": "1"
 }
    
                     
    • 
                 

             

             
For details on formatting log messages into JSON format, see .

         
-        

-            
format-gelf

-            
-            
-            
Syntax:
$(format-gelf)

-            

-            
Description: Available in 3.13 and later.

-            
You can use the Graylog Extended Log Format (GELF) template together with the graylog2() destination to send syslog messages to Graylog. GELF is the native data format of Graylog.

-            

-                
Example: Using the format-gelf template function

-                
The following configuration example shows how you can use the format-gelf template:
destination graylog2 {
-    network(
-        "127.0.0.1"
-        port(12201)
-        transport(tcp)
-        template("$(format-gelf)")
-    );
-};

-            

-        

         

             
format-json

             
@@ -356,9 +171,9 @@ 

             
             
-            
Syntax:
$(format-json parameters)

+            
Syntax:
$(format-json parameters)

             

-            
Description: The format-json template function receives value-pairs as parameters and converts them into JavaScript Object Notation (JSON) format. Including the template function in a message template allows you to store selected information about a log message (that is, its content, macros, or other metadata) in JSON format. Note that the input log message does not have to be in JSON format to use format-json, you can reformat any incoming message as JSON.

+            
Description: The format-json template function receives value-pairs as parameters and converts them into JavaScript Object Notation (JSON) format. Including the template function in a message template allows you to store selected information about a log message (that is, its content, macros, or other metadata) in JSON format. Note that the input log message does not have to be in JSON format to use format-json, you can reformat any incoming message as JSON.

             
You can use the value-pairs that  stores about the log message as JSON fields. Using value-pairs, you can:

             
    
                 
  • 
@@ -372,7 +187,7 @@ 
    
             

             
For details, see Structuring macros, metadata, and other value-pairs. Note that the syntax of format-json is different from the syntax of value-pairs(): format-json uses a syntax similar to command lines.

-            
+            
             
             

                 
Example: Using the format-json template function

@@ -380,19 +195,19 @@ 
Example: Using the format-json template function

   --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
   --pair MSGHDR=\"$PROGRAM[$PID]: \")

The following example shows how to use this template function to store log messages in JSON format:

destination d_json {
-    file(
-        "/var/log/messages.json"
-        template("$(format-json --scope selected_macros --scope nv_pairs)\n")
-    );
+    file("/var/log/messages.json" template("$(format-json --scope selected_macros --scope nv_pairs)\n"));
 };

NOTE: - In the case of syslog-ng macros starting with a dot (for example, ".SDATA.meta.sequenceID"), format-json replaces the dot with an underscore character (for example, {"_SDATA":{"meta":{"sequenceId":"55555"}}}).

-

To retain the starting dot, use the --leave-initial-dot flag, for example:

$(format-json --leave-initial-dot .SDATA.meta.sequenceID)
-
+ In case of syslog-ng macros starting with a dot (for example, ".SDATA.meta.sequenceID") an empty key name is added at the top level of the JSON structure. You can work around this by adding --shift 1 as a parameter to the template function. For example, in case of ".SDATA.meta.sequenceID", an empty key name is added at the top level of the JSON structure:

{"":
+    {"SDATA" :
+        {"meta" :
+            {"sequenceID": "123"}
+        }
+    }
+}
-

If you have to forward your log messages in JSON format, but the receiving application cannot handle nested JSON objects, use the format-flat-json template function. For details, see .

format-welf
@@ -403,124 +218,42 @@

This template function converts value-pairs into the WebTrends Enhanced Log file Format (WELF). The WELF format is a comma-separated list of name=value elements. Note that the order of the elements is random. If the value contains whitespace, it is enclosed in double-quotes, for example, name="value". For details on the WELF format, see https://www3.trustwave.com/support/kb/article.aspx?id=10899.

-

To select which value-pairs to convert, use the command-line syntax of the value-pairs() option. For details on selecting value-pairs, see value-pairs().

+

To select which value-pairs to convert, use the command-line syntax of the value-pairs() option. For details on selecting value-pairs, see value-pairs().

Example: Using the format-welf() template function

The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).

$(format-welf --scope syslog,all_macros,selected_macros \
   --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
   --pair MSGHDR=\"$PROGRAM[$PID]: \")

The following example shows how to use this template function to store log messages in WELF format:

destination d_welf {
-    file(
-        "/var/log/messages.welf"
-        template("$(format-welf --scope selected_macros --scope nv_pairs)\n")
-    );
+    file("/var/log/messages.welf" template("$(format-welf --scope selected_macros --scope nv_pairs)\n"));
 };
-
-
geoip (DEPRECATED)
- - -

This template function is deprecated. Use geoip2 instead.

-

Syntax:

$(geoip <IPv4-address>)
-

-

Description: This template function returns the 2-letter country code of any IPv4 address or host. IPv6 addresses are not supported. Currently only the 2-letter codes are supported, and only from the default database. For example, $(geoip $HOST)

-
-

NOTE: - This template function is available only if has been compiled with the --enable-geoip compiling option.

-
-

To retrieve additional GeoIP information, see Looking up GeoIP data from IP addresses (DEPRECATED).

-
-
+
geoip2
-

Syntax:

$(geoip2 --database <path-to-geoip2-database-file>
+            
Syntax:
$(geoip2 --database <path-to-geoip2-database-file>
     [ --field "registered_country.names.ru" ] ${HOST})

             

-            
Description: This template function extracts specific fields from the mmdb database using the --field parameter. If you omit this parameter, it returns the 2-letter country code of any IPv4/IPv6 address or host.

+            
Description: This template function extracts specific fields from the mmdb database using the --field parameter. If you omit this parameter, it returns the 2-letter country code of any IPv4/IPv6 address or host.

             

                 

                     
NOTE: 
 					This template function is available only if  has been compiled with geoip2 support. To enable it, use the --enable-geoip compiling option.

                 

             

-            
To retrieve additional GeoIP information, see Looking up GeoIP2 data from IP addresses.

+            
To retrieve additional GeoIP information, see Looking up GeoIP2 data from IP addresses.
-
-
getent
- - -

Syntax:

$(getent)
-

-

Description: Available in 3.13 and later.

-

You can use the getent template function to look up entries from the Name Service Switch libraries, such as, passwd, services, or protocols.

-

The following databases are supported:

-
    -
  • -

    passwd -

    -

    Use this database to query data related to a user. Specify the user by either username or user ID. You can query the following data: username, user ID, group ID, GECOS field, home directory, or user shell.

    $(getent passwd testuser name)
-$(getent passwd testuser uid)
-$(getent passwd testuser gid)
-$(getent passwd testuser gecos)
-$(getent passwd testuser dir)
-$(getent passwd testuser shell)
    -

    or

    $(getent passwd 1000 name)
-$(getent passwd 1000 uid)
-$(getent passwd 1000 gid)
-$(getent passwd 1000 gecos)
-$(getent passwd 1000 dir)
-$(getent passwd 1000 shell)
    -

    The queried data is optional. When you do not query any data, the default behavior applies, which is as follows: user ID is returned for username, or username is returned for user ID.

    -
      -
    • -

      Username $(getent passwd testuser) returns user ID 1000.

      -
      • -
    • -

      User ID $(getent passwd 1000) returns username testuser.

      -
      • -
    -
    • -
  • -

    group -

    -

    Use this database to query group-related data. The group can be specified using either group ID or group name. You can query the following data: group name, group ID, and members.

    $(getent group adm name)
-$(getent group adm gid)
-$(getent group adm members)
    -

    The queried data is optional. The default behavior is as follows: group ID is returned for group name, or group name is returned for user ID.

    -
      -
    • -

      Group name $(getent group adm) returns group ID 4.

      -
      • -
    • -

      Group ID $(getent group 4) returns group name adm.

      -
      • -
    -
    • -
  • -

    protocols -

    -

    Use this database to translate protocol name to protocol ID, or protocol ID to protocol string.

    $(getent protocols tcp)
-$(getent protocols 6)
    -
    • -
  • -

    services -

    -

    Use this database to translate service name to service ID, or service ID to service name.

    $(getent services http)
-$(getent services 80)
    -
    • -
-
graphite-output
-

Syntax:

$(graphite-output parameters)
+

Syntax:

$(graphite-output parameters)

-

Description: Available in 3.67.0 and later (Originally appeared in the incubator for syslog-ng 3.5). This template function converts value-pairs from the incoming message to the Graphite plain text protocol format. It is ideal to use with the messages generated by the monitor-source plugin (currently available in the syslog-ng incubator project).

+

Description: Available in 3.67.0 and later (Originally appeared in the incubator for syslog-ng 3.5). This template function converts value-pairs from the incoming message to the Graphite plain text protocol format. It is ideal to use with the messages generated by the monitor-source plugin (currently available in the syslog-ng incubator project).

For details on selecting value-pairs in and for possibilities to specify which information to convert to Graphite plain text protocol format, see Structuring macros, metadata, and other value-pairs. Note that the syntax of graphite-output is different from the syntax of value-pairs(): graphite-output uses a the command-line syntax used in the format-json template function.

Example: Using the graphite-output template function
@@ -533,13 +266,11 @@
Example: Using the graphite-output template function
grep
-

Syntax:

$(grep condition value-to-select)
+

Syntax:

$(grep condition value-to-select)

-

Description: The grep template function can search a message context when correlating messages (for example, when you use a pattern database or the grouping-by parser). The context-lookup template function requires a condition (a filter or a string), and returns a specific macro or template of the matching message (for example, the ${MESSAGE} field of the message).

-
- - -
+

Description: The grep template function can search a message context when correlating messages (for example, when you use a pattern database or the grouping-by parser). The context-lookup template function requires a condition (a filter or a string), and returns a specific macro or template of the matching message (for example, the ${MESSAGE} field of the message).

+ +

You can to specify multiple name-value pairs as parameters, separated with commas. If multiple messages match the condition of grep, these will be returned also separated by commas. This can be used for example, to collect the email recipients from postfix messages.

@@ -554,39 +285,36 @@
hash
-

Syntax:

$(<method> [opts] $arg1 $arg2 $arg3...)
+

Syntax:

$(<method> [opts] $arg1 $arg2 $arg3...)

-

Options:

--length N, -l N
Truncate the hash to the first N characters.

-

Description: Calculates a hash of the string or macro received as argument using the specified hashing method. If you specify multiple arguments, effectively you receive the hash of the first argument salted with the subsequent arguments.

+

Options:

--length N, -l N
Truncate the hash to the first N characters.

+

Description: Calculates a hash of the string or macro received as argument using the specified hashing method. If you specify multiple arguments, effectively you receive the hash of the first argument salted with the subsequent arguments.

<method> can be one of md5, md4, sha1, sha256, sha512 and "hash", which is equivalent to md5. Macros are expected as arguments, and they are concatenated without the use of additional characters.

-

The md4 <method> is deprecated.

This template function can be used for anonymizing sensitive parts of the log message (for example, username) that were parsed out using PatternDB before storing or forwarding the message. This way, the ability of correlating messages along this value is retained.

-

Also, using this template, quasi-unique IDs can be generated for data, using the --length option. This way, IDs will be shorter than a regular hash, but there is a very small possibility of them not being as unique as a non-truncated hash.

+

Also, using this template, quasi-unique IDs can be generated for data, using the --length option. This way, IDs will be shorter than a regular hash, but there is a very small possibility of them not being as unique as a non-truncated hash.

NOTE: These template functions are available only if has been compiled with the --enable-ssl compile option and the tfhash module has been loaded.

- +
Example: Using the $(hash) template function

The following example calculates the SHA1 hash of the hostname of the message:

$(sha1 $HOST)
-

The following example calculates the SHA256 hash of the hostname, using the salted string to salt the result:

$(sha1 $HOST salted)
-

To use shorter hashes, set the --length:

$(sha1 --length 6 $HOST)
+

The following example calculates the SHA256 hash of the hostname, using the salted string to salt the result:

$(sha1 $HOST salted)
+

To use shorter hashes, set the --length:

$(sha1 --length 6 $HOST)

To replace the hostname with its hash, use a rewrite rule:

rewrite r_rewrite_hostname{set("$(sha1 $HOST)", value("HOST"));};
-
- - -
+ +
if
-

Syntax:

$(if (<condition>) <true template> <false template>)
+

Syntax:

$(if (<condition>) <true template> <false template>)

-

Description: Returns the value of the <true template> parameter if the <condition> is true. If the <condition> is false, the value of <false template> is returned.

+

Description: Returns the value of the <true template> parameter if the <condition> is true. If the <condition> is false, the value of <false template> is returned.

Example: Using pattern databases and the if template function

The following example returns violation if the username name-value pair of a message is root, and system otherwise.

$(if ("${username}" == "root") "violation" "system")
@@ -599,62 +327,18 @@
Example: Using pattern databases and the if template function
$(if ("${username}" == "joe") "admin" "normal user"))</value>
-
-
implode
- - -

Syntax:

$(implode <separator> <string1>, <string2>, ...)
-

-

Description: Turns a list into a string combining the pieces together with a separator. You can also use the explode template function, which turns a string separated by a specific character into a list. Available in 3.217 and later.

-
-
Example: Using the implode template function
-

The following configuration example shows how you can use the implode template to turn a list into a string:

- - - - - - - - - - - - - -
ConfigurationResult
$(implode ' ' 'string1,string2,string3,string4,string5')"string1 string2 string3 string4 string5"
-

You can also use a $(list-*) template -function to further manipulate the list. The following example returns the first three elements of the list:

- - - - - - - - - - - - - -
ConfigurationResult
$(implode ' ' $(list-slice :3 string1,string2,string3,string4,string5))"string1 string2 string3"
-
-
indent-multi-line
-

Syntax:

$(indent-multi-line parameter)
+

Syntax:

$(indent-multi-line parameter)

-

Description: This template function makes it possible to write multi-line log messages into a file. The first line is written like a regular message, subsequent lines are indented with a tab, in compliance with RFC822.

+

Description: This template function makes it possible to write multi-line log messages into a file. The first line is written like a regular message, subsequent lines are indented with a tab, in compliance with RFC822.

Example: Using the indent-multi-line template function

The following example writes multi-line messages into a text file.

destination d_file {
-    file (
-        "/var/log/messages"
-        template("${ISODATE} ${HOST} $(indent-multi-line ${MESSAGE})\n")
-    );
+        file ("/var/log/messages"
+                template("${ISODATE} ${HOST} $(indent-multi-line ${MESSAGE})\n") );
 };
@@ -662,16 +346,17 @@
Example: Using the indent-multi-line template function
ipv4-to-int
-

Syntax:

$(ipv4-to-int parameter)
+

Syntax:

$(ipv4-to-int parameter)

-

Description: Converts the specified IPv4 address to its numeric representation. The numerical value of an IPv4 address is calculated by treating the IP address as a 4-byte hexadecimal value. For example, the 192.168.1.1 address equals to: 192=C0, 168=A8, 1=01, 1=01, or C0A80101, which is 3232235777 in decimal representation.

+

Description: Converts the specified IPv4 address to its numeric representation. The numerical value of an IPv4 address is calculated by treating the IP address as a 4-byte hexadecimal value. For example, the 192.168.1.1 address equals to: 192=C0, 168=A8, 1=01, 1=01, or C0A80101, which is 3232235777 in decimal representation.

-

NOTE: This template function is available only if the convertfuncs module has been loaded.

- +

NOTE: + This template function is available only if the convertfuncs module has been loaded.

+
-
+
List manipulation

The list-* template functions allow you to manipulate comma-separated lists. Such lists represent a simple array type in . Note the following about formatting lists:

    @@ -686,140 +371,95 @@
    List mani

These template functions return a well-formed list, properly encoding and quoting all elements. If a template function returns a single element, all quotation is decoded and the value contains the literal value.

-

Starting with version 3.10, the following list-related template functions are available. Certain functions allow you to reference an element using its number: note that the list index starts with zero, so the index of the first element is 0, the second element is 1, and so on.

-
-
list-append +

Starting with version 3.107.0.15, the following list-related template functions are available. Certain functions allow you to reference an element using its number: note that the list index starts with zero, so the index of the first element is 0, the second element is 1, and so on.

+
+
list-append
-

Syntax:

$(list-append ${list} ${name-value-pair1} ${name-value-pair2} ... )
+

Syntax:

$(list-append ${list} ${name-value-pair1} ${name-value-pair2} ... )

-

Description: Returns a list and appends the values of the specified name-value pairs to the end of the list. You can also append elements to an empty list, for example, $(list-append '' 'element-to-add')

+

Description: Returns a list and appends the values of the specified name-value pairs to the end of the list. You can also append elements to an empty list, for example, $(list-append '' 'element-to-add')

-
-
list-concat +
+
list-concat
-

Syntax:

$(list-concat ${name-value-pair1} ${name-value-pair2} ... )
The commas between the parameters are optional.

-

Description: This template function creates (concatenates) a list of the values it receives as parameter. The values can be single values (for example, ${HOST}) or lists.

+

Syntax:

$(list-concat ${name-value-pair1} ${name-value-pair2} ... )
The commas between the parameters are optional.

+

Description: This template function creates (concatenates) a list of the values it receives as parameter. The values can be single values (for example, ${HOST}) or lists.

For example, the value of the $(list-concat ${HOST}, ${PROGRAM}, ${PID}) is a comma-separated list.

You can concatenate existing lists into a single list using:

$(list-concat ${list1} ${list2})
-
-
list-count +
+
list-count
-

Syntax:

$(list-count ${list} )
+

Syntax:

$(list-count ${list} )

-

Description: Returns the number of elements in the list.

+

Description: Returns the number of elements in the list.

-
-
list-head +
+
list-head
-

Syntax:

$(list-head ${list} )
+

Syntax:

$(list-head ${list} )

-

Description: Returns the first element of the list, unquoted.

+

Description: Returns the first element of the list, unquoted.

-
-
list-nth +
+
list-nth
-

Syntax:

$(list-nth <index-number> ${list} )
+

Syntax:

$(list-nth <index-number> ${list} )

-

Description: Returns the nth element of the list, unquoted. Note that the list index starts with zero, so (list-nth 1 ${list} ) returns the second element, and so on.

+

Description: Returns the nth element of the list, unquoted. Note that the list index starts with zero, so (list-nth 1 ${list} ) returns the second element, and so on.

-
-
list-search +
+
list-tail
- + -

Syntax:

$(list-search [OPTIONS] <pattern> ${list})
+

Syntax:

$(list-tail ${list} )

-

Description: The list-search template function searches the elements of ${list} starting at the specified start_index, then returns the index of the first match of <pattern> within ${list}.

-
-

NOTE: Indexing is 0-based. If <pattern> is not found, the function returns an empty string.

-
-

Options:

-
    -
  • --mode MODE: Matching mode, with the following possible values: literal (default), prefix, substring, glob, pcre
    • -
  • --start-index N: Skips N elements in the ${list}
    • -
+

Description: Returns the list without the first element. For example, if the ${mylist} list contains the one, two, three elements, then $(list-tail ${mylist} ) returns two, three.

-
-
list-slice +
+
list-slice
-

Syntax:

$(list-slice <from>:<to> ${list} )
+

Syntax:

$(list-slice <from>:<to> ${list} )

-

Description: Returns the specified subset of the list. Note that the list index starts with zero, for example, $(list-slice 1:2 ${list} ) returns the second and third element of the list, and so on.

+

Description: Returns the specified subset of the list. Note that the list index starts with zero, for example, $(list-slice 1:2 ${list} ) returns the second and third element of the list, and so on.

You can omit the from or to index if you want to start the subset from the beginning or end of the list, for example: 3: returns the list starting with the 4th element, while :3 returns the first four elements.

Negative numbers select an element from the end of the list, for example, -3: returns the last three element of the list.

-
-
list-tail -
- - -

Syntax:

$(list-tail ${list} )
-

-

Description: Returns the list without the first element. For example, if the ${mylist} list contains the one, two, three elements, then $(list-tail ${mylist} ) returns two, three.

-
-
-
length
+
+
length
-

Syntax:

$(length "<macro>")
+

Syntax:

$(length "<macro>")

-

Description: Returns the length of the macro in characters, for example, the length of the message. For example, the following filter selects messages that are shorter than 16 characters:

f_short {
+            
Description: Returns the length of the macro in characters, for example, the length of the message. For example, the following filter selects messages that are shorter than 16 characters:
f_short {
     match ('-', value ("$(if ($(length "${MESSAGE}") <= 16) "-" "+")"));
 };
-
-
lowercase
+
+
lowercase
-

Syntax:

$(lowercase "<macro>")
+

Syntax:

$(lowercase "<macro>")

-

Description: Returns the lowercase version of the specified string or macro. For example, the following example uses the lowercase version of the hostname in a directory name:

destination d_file {
+            
Description: Returns the lowercase version of the specified string or macro. For example, the following example uses the lowercase version of the hostname in a directory name:
destination d_file {
     file ("/var/log/${MONTH}/${DAY}/$(lowercase "${HOST}")/messages");
 };

             
Available in 3.57.0 and later.
-
-
map
- -

Syntax:

$(map template list)
-

-

Description: Returns with a list that contains the results of applying a template function for each elements of a list.

-

Available in version 3.28 and later.

-

Parameters: -

-
    -
  • -

    template: Mandatory. This template function is applied for each elements of the list. Use $_ macro to refer to the current list element.

    -
    • -
  • -

    list: Mandatory. A list, or template.

    -
    • -
-

Example: -

-

When used in configuration as seen in the example, the map template function adds one to each element of a list:

log {
-  source { example-msg-generator(num(1) values(LST => "0,1,2")); };
-  destination {
-    file("/dev/stdout"template('$(map "$(+ 1 $_)" $LST)')
-    );
-  };
-};
-

The returned values are 1, 2, and 3.

-
Numerical operations
@@ -840,41 +480,26 @@
-

Syntax:

$(<operation> "<value1>" "<value2>")
+

Syntax:

$(<operation> "<value1>" "<value2>")

-

Description: These template functions allow you to manipulate numbers, that is, to perform addition (+), substraction (-), multiplication (*), division (/), and modulus (%). All of them require two numeric arguments. The result is NaN (Not-a-Number) if the parameters are not numbers, cannot be parsed, or if a division by zero would occur. For example, to add the value of two macros, use the following template function:

$(+ "${<MACRO1>}" "${<MACRO2>}");
-

Starting with version 3.227.0.15 and later, the numerical operators support floating-point values. They behave like the operators in the C programming language:

-
    -
  • -

    If both operands are integers, they return an integer.

    -
    • -
  • -

    If any of the operands is a floating-point number, they return a floating-point result.

    -
    • -
-

For example:

$(/ 3 2) # Both operands are integers, so the result is 1
-# One of the operands is a floating point number, so the result is also floating-point
-$(/ 3.0 2) # = 1.500000
-$(/ 3 2.0) # = 1.500000
-$(/ 3.0 2.0) # = 1.500000
-

To round floating-point numbers, you can use the ceil, floor, and round template functions.

+

Description: These template functions allow you to manipulate numbers, that is, to perform addition (+), substraction (-), multiplication (*), division (/), and modulus (%). All of them require two numeric arguments. The result is NaN (Not-a-Number) if the parameters are not numbers, cannot be parsed, or if a division by zero would occur. For example, to add the value of two macros, use the following template function:

$(+ "${<MACRO1>}" "${<MACRO2>}");
-

When you are correlating messages and a name-value pair contains numerical values in the messages, you can calculate the lowest (min), highest (max), total (sum), and mean (average) values. These calculations process every message of the correlation context. For details on message correlation, see Correlating log messages. For example, if the messages of the context have a .myfields.load name-value pair, you can find the highest load value using the following template function.

$(max ${.myfields.load})
+

When you are correlating messages and a name-value pair contains numerical values in the messages, you can calculate the lowest (min), highest (max), total (sum), and mean (average) values. These calculations process every message of the correlation context. For details on message correlation, see Correlating log messages. For example, if the messages of the context have a .myfields.load name-value pair, you can find the highest load value using the following template function.

$(max ${.myfields.load})
or
-

Syntax:

$(or <macro1> <macro2>)
+

Syntax:

$(or <macro1> <macro2>)

-

Description: This template function returns the first non-empty argument.

+

Description: This template function returns the first non-empty argument.

padding
-

Syntax:

 $(padding <macro> <width> <prepended-character-or-string>)
+

Syntax:

 $(padding <macro> <width> <prepended-character-or-string>)

-

Description: This template function returns the value of its first parameter (a string or macro), prepended with a string. This string is <width> long, and repeats the character or string set in the third parameter. If you use a single character, it is added <width> times. If you use a string, it is repeated until its length reaches <width>. The default padding character is ' ' (space). For example:

+

Description: This template function returns the value of its first parameter (a string or macro), prepended with a string. This string is <width> long, and repeats the character or string set in the third parameter. If you use a single character, it is added <width> times. If you use a string, it is repeated until its length reaches <width>. The default padding character is ' ' (space). For example:

Example: Using the padding template function

If the value of the ${MESSAGE} macro is mymessage, then the output of the padding() template function is the following:

$(padding ${MESSAGE} 10 X)
@@ -884,9 +509,9 @@
Example: Using the padding template function
python
-

Syntax:

 $(python <name-of-the-python-method-to-use> <arguments-of-the-method>)
+

Syntax:

 $(python <name-of-the-python-method-to-use> <arguments-of-the-method>)

-

Description: This template function enables you to write a custom template function in Python. You can define a Python block in your configuration file, define one or more Python functions in it, and use the methods as template functions. If you use a Python block, embeds a Python interpreter to process the messages.

+

Description: This template function enables you to write a custom template function in Python. You can define a Python block in your configuration file, define one or more Python functions in it, and use the methods as template functions. If you use a Python block, embeds a Python interpreter to process the messages.

The following points apply to Python parsers.

@@ -897,18 +522,14 @@
pytho

  • The value of the template function is return value of the Python function.

    • -
  • -

    To reference a name-value pair or a macro in the Python function, use the dot-notation. For example, if the first argument in the definition of the function is called log-message, the value of the HOST macro is log-message.HOST, and so on.

    -
    • -
  • -

    You can define new name-value pairs in the Python function. For example, if the first argument in the definition of the function is called log-message, you can create a new name-value pair like this: log_message["new-macro-name"]="value". This is useful when you parse a part of the message from Python, or lookup a value based on data extracted from the log message.

    -
    • + +
    -
    Declaration:
    python {
-def <name_of_the_python_function>(<log_message>, <optional_other_arguments>):
-    # <your-python-code>
-    return <value_of_the_template_function>
+                
    Declaration
    python {
+    def <name_of_the_python_function>(<log_message>, <optional_other_arguments>):
+        # <your-python-code>
+        return <value_of_the_template_function>
 };
 
 template <template-name> {
@@ -917,33 +538,30 @@ 
    Declaration:
    python {
    Example: Writing template functions in Python
    -

    The following example creates a Python template function called return_message that returns the MESSAGE part of the log message.

    @version: 
+                
    The following example creates a Python template function called return_message that returns the MESSAGE part of the log message.
    @version: 
 
 python {
-def return_message(log_message):
-    return log_message.MESSAGE
+    def return_message(log_message):
+        return log_message['MESSAGE']
 };
 
 destination d_local {
     file("/tmp/logs.txt" template("[$(python return_message)]\n"));
 };
    
-                
    The following example creates a Python template function called resolve_host that receives an IP address as an argument, and attempts to resolve it into a hostname.
    @version: 
+                
    The following example creates a Python template function called resolve_host that receives an IP address as an argument, and attempts to resolve it into a hostname.
    @version: 
 
 python {
-import socket
+    import socket
 
-def resolve_host(log_message, hostname):
-    try:
-        return socket.gethostbyaddr(hostname)[0]
-    except (socket.herror, socket.error):
-        return 'unknown'
+    def resolve_host(log_message, hostname):
+        try:
+            return socket.gethostbyaddr(hostname)[0]
+        except (socket.herror, socket.error):
+            return 'unknown'
 };
 
 destination d_local {
-    file(
-        "/tmp/logs.txt"
-        template("${ISODATE} $(python resolve_host ${SOURCE_IP}) ${MESSAGE}\n")
-    );
+    file("/tmp/logs.txt" template("${ISODATE} $(python resolve_host ${SOURCE_IP}) ${MESSAGE}\n"));
 };
    @@ -951,27 +569,18 @@
    replace-delimiter
    -

    Syntax:

    $(replace-delimiter "<old-delimiters>" "<new-delimiter>" "<macro>")
    +

    Syntax:

    $(replace-delimiter "<old-delimiters>" "<new-delimiter>" "<macro>")

    -

    Description: Replaces the delimiter character with a new one. For example, the following example replaces the tabulators (\t) in the message with semicolons (;):

    $(replace-delimiter "\t" ";" "${MESSAGE}")
    +

    Description: Replaces the delimiter character with a new one. For example, the following example replaces the tabulators (\t) in the message with semicolons (;):

    $(replace-delimiter "\t" ";" "${MESSAGE}")

    Available in 3.57.0 and later.

    -
    -
    round
    - - -

    Syntax:

    $(round argument)
    -

    -

    Description: Rounds a floating-point number to the nearest integer. For example, $(round 1.5) is 2. See also the ceil and floor template functions.

    -

    This template function has an optional second argument that sets the precision of rounding. The default is 0 (output a natural number), but values up to 20 are accepted. For example, $(round 2.123456 4) is 2.1235.

    -
    sanitize
    -

    Syntax:

    $(sanitize <options> "<macro1>" "<macro2> ...")
    +

    Syntax:

    $(sanitize <options> "<macro1>" "<macro2> ...")

    -

    Description: This file replaces the special characters in macro values, for example, it can replace the slash (/) characters in a filename with the underscore (_) character. If you specify multiple arguments, they will be concatenated using the / character, so they can be used as separate directory levels when used in filenames.

    +

    Description: This file replaces the special characters in macro values, for example, it can replace the slash (/) characters in a filename with the underscore (_) character. If you specify multiple arguments, they will be concatenated using the / character, so they can be used as separate directory levels when used in filenames.

    The function has the following options:

    • --ctrl-chars or -c @@ -1001,29 +610,21 @@
      Example: Using the sanitize template function

      This is equivalent to file("/var/log/$HOST/$PROGRAM/messages");, but any slashes in the values of the $HOST and $PROGRAM macros are replaced with underscores.

    -
    -
    stardate
    - - -

    Syntax:

    $(stardate [option] "<date-in-unixtime>")
    -

    -

    Description: Converts a date in UNIXTIME (for example, ${UNIXTIME}) into stardate, displaying the year and the progress of the year in a number of digits (YYYY.NNN). You can set the number of digits using the --digits option, for example:

    $(stardate --digits 2 "${R_UNIXTIME}")
    -
    strip
    -

    Syntax:

    $(strip "<macro>")
    +

    Syntax:

    $(strip "<macro>")

    -

    Description: Deletes whitespaces from the beginning and the end of a macro. You can specify multiple macros separated with whitespace in a single template function, for example:

    $(strip "${MESSAGE}" "${PROGRAM}")
    +

    Description: Deletes whitespaces from the beginning and the end of a macro. You can specify multiple macros separated with whitespace in a single template function, for example:

    $(strip "${MESSAGE}" "${PROGRAM}")
    substr
    -

    Syntax:

    $(substr "<argument>" "<offset>" "<length>")
    +

    Syntax:

    $(substr "<argument>" "<offset>" "<length>")

    -

    Description: This function extracts a substring of a string.

    +

    Description: This function extracts a substring of a string.

    • argument
    • @@ -1032,12 +633,12 @@
      subst
    • offset
    • -

      Specifies where the substring begins (in characters). 0 means to start from the beginning of the string, 5 means to skip the first 5 characters of the string, and so on. Use negative numbers to specify where to start from the end of the string, for example, -1 means the last character, -5 means to start five characters before the end of the string.

      +

      Specifies where the substring begins (in characters). 0 means to start from the beginning of the string, 5 means to skip the first 5 characters of the string, and so on. Use negative numbers to specify where to start from the end of the string, for example, -1 means the last character, -5 means to start five characters before the end of the string.

    • length
    • -

      Optional parameter: The number of characters to extract. If not specified, the substring will be extracted from the offset to the end of the string. Use negative numbers to stop the substring before the end of the string, for example, -5 means the substring ends five characters before the end of the string.

      +

      (Optional parameter): The number of characters to extract. If not specified, the substring will be extracted from the offset to the end of the string. Use negative numbers to stop the substring before the end of the string, for example, -5 means the substring ends five characters before the end of the string.

    @@ -1052,36 +653,24 @@
    Example: Using the substr template function
    };
    -
    -
    template
    - - -

    Syntax:

    $(template <template-name>)
-$(template $<dynamic-template-name>)
-$(template $<dynamic-template-name> '<optional-fallback-template>')
    -

    -

    Description: This template function looks up the <template-name> in the configuration and uses that to format its result. The referenced template can be static or dynamic. For static templates, resolves the template when it starts, or when the configuration is reloaded. For dynamic templates, the results are resolved runtime (for dynamic templates, the template name contains at least one '$' character). For example, the name of the template to be invoked can be extracted from the message, or from a name-value pair set using the add-contextual-data() feature.

    -

    For dynamic templates, you can set an optional second template. This second template will be the results of the template function if resolving the dynamic template fails for some reason. For example:

    $(template ${my-dynamic-template} '$DATE $HOST $MSGHDR$MSG\n')
    -

    Available in 3.227.0.15 and later.

    -
    uppercase
    -

    Syntax:

    $(uppercase "<macro>")
    +

    Syntax:

    $(uppercase "<macro>")

    -

    Description: Returns the uppercase version of the specified string or macro. For example, the following example uses the uppercase version of the hostname in a directory name:

    destination d_file {
-    file ("/var/log/${MONTH}/${DAY}/$(uppercase "${HOST}")/messages");
+            
    Description: Returns the uppercase version of the specified string or macro. For example, the following example uses the uppercase version of the hostname in a directory name:
    destination d_file {
+file ("/var/log/${MONTH}/${DAY}/$(uppercase "${HOST}")/messages");
 };
    
-            
    Available in 3.57.0 and later.
    
+            
    Available in 3.57.0 and later.
    url-decode
    -

    Syntax:

    $(url-decode <string-pr-macro-1> <string-pr-macro-2> ... )
    +

    Syntax:

    $(url-decode <string-pr-macro-1> <string-pr-macro-2> ... )

    -

    Description: You can use the url-decode template function to decode url-encoded strings and macros. For example, $(url-decode %3C%3E) yields <>. The url-decode can receive multiple parameters (maximum 64). In this case, each parameter is decoded separately, and simply concatenated.

    +

    Description: You can use the url-decode template function to decode url-encoded strings and macros. For example, $(url-decode %3C%3E) yields <>. The url-decode can receive multiple parameters (maximum 64). In this case, each parameter is decoded separately, and simply concatenated.

    Available in version 3.187.0.11 and later.

    @@ -1090,44 +679,41 @@
    url-encode -

    Syntax:

    $(url-encode ${MESSAGE} )\n")
    +

    Syntax:

    $(url-encode ${MESSAGE} )\n")

    -

    Description: You can use the url-encode template function together with the telegram() destination to send syslog messages to Telegram. The url-encode template function escapes strings. All input characters that are not a-z, A-Z, 0-9, '-', '.', '_' or '~' are converted to their "URL escaped" version.

    +

    Description: You can use the url-encode template function together with the telegram() destination to send syslog messages to Telegram. The url-encode template function escapes strings. All input characters that are not a-z, A-Z, 0-9, '-', '.', '_' or '~' are converted to their "URL escaped" version.

    Available in version 3.187.0.11 and later. (In version 3.16-3.17, this template function was called urlencode.)

    uuid
    -

    Syntax:

    $(uuid)
    +

    Syntax:

    $(uuid)

    -

    Description: Generates a Universally Unique IDentifier (UUID) that complies with RFC4122. That way, an UUID can be added to the message soon after it is received, so messages stored in multiple destinations can be identified. For example, when storing messages in a database and also in files, the UUID can be used to find a particular message both in the database and the files.

    +

    Description: Generates a Universally Unique IDentifier (UUID) that complies with RFC4122. That way, an UUID can be added to the message soon after it is received, so messages stored in multiple destinations can be identified. For example, when storing messages in a database and also in files, the UUID can be used to find a particular message both in the database and the files.

    To generate a UUID, you can use a rewrite rule to create a new value-pair for the message.

    Example: Using Universally Unique Identifiers
    -

    The following example adds a value-pair called MESSAGE_UUID to the message using a rewrite rule and a template.

    rewrite r_add_uuid {
-    set("$(uuid)" value("MESSAGE_UUID"));
-};
+                
    The following example adds a value-pair called MESSAGE_UUID to the message using a rewrite rule and a template.
    
+            
    rewrite r_add_uuid { set("$(uuid)" value("MESSAGE_UUID")); };
 
 destination d_file {
-    file (
-        "/var/log/messages"
-        template("$MESSAGE_UUID $ISODATE $HOST $MSG\n")
-        template-escape(no)
+    file ("/var/log/messages"
+          template("$MESSAGE_UUID $ISODATE $HOST $MSG\n")
+          template-escape(no)
     );
 };
 
-log {
-    source(s_network);
-    rewrite(r_add_uuid);
-    destination(d_file);
+log { source(s_network);
+      rewrite(r_add_uuid);
+      destination(d_file);
 };
    -
    -

    NOTE: This template function is available only if the tfuuid module has been loaded.

    - +

    NOTE: + This template function is available only if the tfuuid module has been loaded.

    +
    - + \ No newline at end of file diff --git a/Content/Guides/syslog-ng-guide-admin/specifying-data-types.htm b/Content/Guides/syslog-ng-guide-admin/specifying-data-types.htm index ef6cb7b04..df100f682 100644 --- a/Content/Guides/syslog-ng-guide-admin/specifying-data-types.htm +++ b/Content/Guides/syslog-ng-guide-admin/specifying-data-types.htm @@ -1,10 +1,11 @@  - + + -

    Specifying data types in value-pairs

    +

    Specifying data types in value-pairs

    @@ -14,6 +15,7 @@

    Specifying data types in value-pairs

    By default, handles every data as strings. However, certain destinations and data formats (for example, SQL, MongoDB, JSON, AMQP) support other types of data as well, for example, numbers or dates. The application allows you to specify the data type in templates (this is also called type-hinting). If the destination driver supports data types, it converts the incoming data to the specified data type. For example, this allows you to store integer numbers as numbers in MongoDB, instead of strings.

    +

    From version 4.0 onwards, name-value pairs are now triplets (name, type, value). Typing support is available for several other components, for example, json-parser() and the $(format-json) template function. For more information, see Components supported by data types.

    @@ -27,26 +29,27 @@

    Specifying data types in value-pairs

    Caution: -

    Hazard of data loss! If cannot convert the data into the specified type, an error occurs, and drops the message by default. To change how handles data-conversion errors, see Global options.

    +

    Hazard of data loss! If cannot convert the data into the specified type, an error occurs, and drops the message by default. To change how handles data-conversion errors, see on-error().
    -

    To use type-hinting, enclose the macro or template containing the data with the type: <datatype>("<macro>"), for example: int("$PID").

    -

    Currently the mongodb() destination and the format-json and format-flat-json() template functions support data types.

    +

    To use type-hinting, enclose the macro or template containing the data with the type: <datatype>("<macro>"), for example: int("$PID").

    +

    Currently the mongodb() destination and the format-json template function supports data types.

    Example: Using type-hinting

    The following example stores the MESSAGE, PID, DATE, and PROGRAM fields of a log message in a MongoDB database. The DATE and PID parts are stored as numbers instead of strings.

    mongodb(
-    value-pairs(pair("date", datetime("$UNIXTIME"))
+    value-pairs(
+        pair("date", datetime("$UNIXTIME"))
         pair("pid", int64("$PID"))
         pair("program", "$PROGRAM"))
         pair("message", "$MESSAGE"))
     )
 );
    -

    The following example formats the same fields into JSON.

    $(format-json date=datetime($UNIXTIME) pid=int64($PID) program=$PROGRAM message=$MESSAGE)
    -

    The following example formats the MESSAGE field as a JSON list.

    $(format-json message=list($MESSAGE))"
    +

    Use the following example to format the same fields into JSON.

    $(format-json date=datetime("$UNIXTIME") pid=int64("$PID") program="$PROGRAM" message="$MESSAGE")
    +

    Use the following example to format the MESSAGE field as a JSON list.

    $$(format-json message=list($MESSAGE))

    The application currently supports the following data-types.

      @@ -62,18 +65,404 @@
      Example: Using type-hinting

    • literal: The data as a literal string, without adding any quotes or escape characters.

      • -
    • -

      list: The data as a list. For details, see the list manipulation template functions in Template functions of [%=General.OSE%].

      -

    • int or int32: 32-bit integer.

    • -

      int64: 64-bit integer.

      +

      int64: 64-bit integer.

    • -

      string: The data as a string.

      +

      string: The data is a string.

      +
      • +
    • +

      list: The data is a list of strings.

      +
      • +
    • +

      JSON: The data is a JSON snippet.

      +
      • +
    • +

      null: The type of the data is undefined.

    +
    +
    Components supported by data types
    +

    The following components support data types from 4.0 and onwards:

    +
    +

    NOTE:

    +

    Component types not listed below process data as string.

    +
    +
      +

    • Numeric operators in filter expression comparisons are now type-aware. The exact comparison depends on the types associated with the values compared. For more information, see Comparing macro values in filters.

      • +

    • json-parser() and the format-json template function:

      +

      For more information, see json-parser() and format-json.

      +

      converts all elements in a JSON object to name-value pairs, when using json-parser(). Any type related data present in the original JSON is retained. This data is propagated automatically to any other component that supports type, for example a destination.

      +

      Elements without type data are handled as strings.

      +

      JSON lists (arrays) are converted to lists, and can be manipulated using the $(list-*) + template functions.

      +
      • +

    • set() and groupset() rewrite rules: +

      +

      The type of the field can be set. Type-casting can be executed using the set() and groupset() template functions, to properly promote the type information.

      +

      For more information, see set() and groupset().

      +
      • +

    • db-parser():

      +

      db-parser() rules can pair types with values using the type attribute.

      + +
      +
      Example: Using the type attribute
      +

      <value name="foobar" type="integer">$PID</value>

      +
      +

      The integer is a type-cast that couples $foobar with an integer type. The internal parsers of db-parser() (for example, @NUMBER@) automatically couple type information to the parsed name-value pair.

      +

      For more information, see db-parser().

      +
      • +

    • add-contextual-data(): Name-value pairs that are populated using add-contextual-data() propagate type information, similarly to db-parser().

      +
      • +

    • map-value-pairs():

      +

      map-value-pairs() propagates type information.

      +
      • +

    • SQL type support:

      +

      Columns with specific type information are stored with this information kept intact. For more information, see sql(): Storing messages in an SQL database. +

      +
      • +

    • Template type support:

      +

      Templates can be cast explicitly to a specific type. Templates also propagate type information from macros, template functions, and values in the template string.

      +
      • +

    • python() typing:

      +

      Python components (sources, destinations, parsers, and template functions) support all data types, except for json().

      +
      • +

    • On-disk serialized formats (that is, disk buffer):

      +

      Version 4.0 and newer versions are backwards compatible with messages serialized with earlier versions, and the format is compatible for downgrades. Therefore, even if a newer version of serialized a message, older versions and associated tools are able to read it, however, in this case the type information is lost.

      +
      • +
    +
    value-pairs()
    +

    + + +

    + + + + + + + + + + + + + +
    Type:parameter list of the value-pairs() option
    Default:
    empty string
    +
    +

    Description: The value-pairs() option allows you to select specific information about a message easily using predefined macro groups. The selected information is represented as name-value pairs and can be used formatted to JSON format, or directly used in a mongodb() destination.

    +
    +
    Example: Using the value-pairs() option
    + + +

    The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message.

    value-pairs(
+    scope(nv_pairs core syslog all_macros selected_macros everything)
+    exclude("R_*")
+    exclude("S_*")
+    key(".SDATA.meta.sequenceId")
+    pair("MSGHDR" "$PROGRAM[$PID]: ")
+)
    +

    The following example selects the same information as the previous example, but converts it into JSON format.

    $(format-json --scope nv_pairs,core,syslog,all_macros,selected_macros,everything \
+  --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
+  --pair MSGHDR="$PROGRAM[$PID]: ")
    +
    +
    +

    NOTE: + Every macro is included in the selection only once, but redundant information may appear if multiple macros include the same information (for example, including several date-related macros in the selection).

    +
    +

    The value-pairs() option has the following parameters. The parameters are evaluated in the following order:

    +
      +
    1. +

      scope() +

      +
      2. +
    2. +

      exclude() +

      +
      3. +
    3. +

      key() +

      +
      4. +
    4. +

      pair() +

      +
      5. +
    + + + + + + + + + + + + + + + + + + + + +
    + + exclude() + +

     

    +
    Type:Space-separated list of macros to remove from the selection created using the scope() option.
    Default: + + empty string
    +

    Description: This option removes the specified macros from the selection. Use it to remove unneeded macros selected using the scope() parameter.

    +

    For example, the following example removes the SDATA macros from the selection.

    value-pairs(
+    scope(rfc5424 selected_macros)
+    exclude(".SDATA*")
+)
    +

    The name of the macro to remove can include wildcards (*, ?). Regular expressions are not supported.

    +
    + + + + + + + + + + + + + + + + + + + + +
    key() + + + +

     

    +
    Type:Space-separated list of macros to be included in selection
    Default:empty string
    +

    Description: This option selects the specified macros. The selected macros will be included as MACRONAME = MACROVALUE, that is using key("HOST") will result in HOST = $HOST. You can use wildcards (*, ?) to select multiple macros. For example:

    value-pairs(
+    scope(rfc3164)
+    key("HOST")
+)
    value-pairs(
+    scope(rfc3164)
+    key("HOST", "PROGRAM")
+)
    +
    + + + + + + + + + + + + + + + + + + + + + + +
    omit-empty-values() + + + +

     

    +
    Type:flag
    Default:N/A
    +

    Description: If this option is specified, does not include value-pairs with empty values in the output. For example: $(format-json --scope none --omit-empty-values) or

    value-pairs(
+    scope(rfc3164 selected-macros)
+    omit-empty-values()
+)
    +

    Available in version 7.0.143.21 and later.

    +
    + + + + + + + + + + + + + + + + + + + + + + +
    pair() + + + +

     

    +
    Type:name value pairs in "<NAME>" "<VALUE>" format
    Default:empty string
    +

    Description: This option defines a new name-value pair to be included in the message. The value part can include macros, templates, and template functions as well. For example:

    +
    + + + + + + + + + + + + + + + + +

    The following sample selects every value-pair that begins with .cee., deletes this prefix by cutting 4 characters from the names, and adds a new prefix (events.).

    value-pairs(
+    key(".cee.*"
+        rekey(
+            shift(4)
+            add-prefix("events.")
+        )
+    )
+)
    +

    The rekey() option can be used with the format-json template-function as well, using the following syntax:

    $(format-json --rekey .cee.* --add-prefix events.)
    + + + + + + + + +
    rekey() + + + +

     

    +
    Type:<pattern-to-select-names>, <list of transformations>
    Default:empty string
    +

    Description: This option allows you to manipulate and modify the name of the value-pairs. You can define transformations, which are are applied to the selected name-value pairs. The first parameter of the rekey() option is a glob pattern that selects the name-value pairs to modify. If you omit the pattern, the transformations are applied to every key of the scope. For details on globs, see glob.

    +
      +
    • +

      If rekey() is used within a key() option, the name-value pairs specified in the glob of the key() option are transformed.

      +
      • +
    • +

      If rekey() is used outside the key() option, every name-value pair of the scope() is transformed.

      +
      • +
    +

    The following transformations are available:

    +
      +
    • add-prefix("<my-prefix>") + + +
    • +

      Adds the specified prefix to every name. For example, rekey( add-prefix("my-prefix."))

      +
      • + +
    • replace-prefix("<prefix-to-replace>", "<new-prefix>") + + +
    • +

      Replaces a substring at the beginning of the key with another string. Only prefixes can be replaced. For example, replace-prefix(".class", ",patterndb") changes the beginning tag .class to .patterndb

      +
      • + +
    • shift("<number>") + + +
    • +

      Cuts the specified number of characters from the beginning of the name.

      +
      • + +
    +
    +
    Example:
    Using the rekey() option
    + + + + + + + + + + + + + + + + + + + + +
    scope() + + + +

    +

    +
    Type:space-separated list of macro groups to include in selection
    Default:empty string
    +

    Description: This option selects predefined groups of macros. The following groups are available:

    +
      +
    • +

      nv-pairs: Every soft macro (name-value pair) associated with the message, except the ones that start with a dot (.) character. Macros starting with a dot character are generated within and are not originally part of the message, therefore are not included in this group.

      +
      • +
    • +

      dot-nv-pairs: Every soft macro (name-value pair) associated with the message which starts with a dot (.) character. For example, .classifier.rule_id and .sdata.*. Macros starting with a dot character are generated within and are not originally part of the message.

      +
      • +
    • +

      all-nv-pairs: Include every soft macro (name-value pair). Equivalent to using both nv-pairs and dot-nv-pairs.

      +
      • +
    • +

      rfc3164: The macros that correspond to the RFC3164 (legacy or BSD-syslog) message format: $FACILITY, $PRIORITY, $HOST, $PROGRAM, $PID, $MESSAGE, and $DATE.

      +
      • +
    • +

      rfc5424: The macros that correspond to the RFC5424 (IETF-syslog) message format: $FACILITY, $PRIORITY, $HOST, $PROGRAM, $PID, $MESSAGE, $MSGID, $R_DATE, and the metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with .SDATA..

      +

      The rfc5424 group also has the following alias: syslog-proto. Note that the value of $R_DATE will be listed under the DATE key.

      +

      The rfc5424 group does not contain any metadata about the message, only information that was present in the original message. To include the most commonly used metadata (for example, the $SOURCEIP macro), use the selected-macros group instead.

      +
      • +
    • +

      all-macros: Include every hard macro. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).

      +
      • +
    • +

      selected-macros: Include the macros of the rfc3164 groups, and the most commonly used metadata about the log message: the $TAGS, $SOURCEIP, and $SEQNUM macros.

      +
      • +
    • +

      sdata: The metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with .SDATA.

      +
      • +
    • +

      everything: Include every hard and soft macros. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).

      +
      • +
    +

    For example:

    value-pairs(
+    scope(rfc3164 selected-macros)
+)
    +
    +
    \ No newline at end of file