filtering empty fields

[dannygueta]

Hi,

we are using log 2 metric and we are trying to replace a specific field in a json if its null/empty, here is an example log line:

{"hostname":"xxxxxxx","method":"POST","path":"YYYYYYY","status":"200","request_time":"0.000","partner":"111","cache_status":"HIT","nginx_host":"knginx-785bf9c75f-5mmbq","error_output":""}

what we'd like to do is if error_output is empty we want to replace it with 'OK' if it contains an error we want to keep the error, we've tried multiple options, none seem to work properly, this is what we have been trying:

[transforms.filter_notempty]
type = "filter"
inputs = ["nginx_source_metrics"]
condition = ".error_output != """

[transforms.filter_empty]
type = "filter"
inputs = ["nginx_source_metrics"]
condition = ".error_output == """

[transforms.empty_remap]
type = "remap" # required
inputs = ["filter_empty"] # required
source = '''
structured, err = parse_json(.message)
if err != null {
log("Unable to parse_json RAW:" + string!(.message), level: "error")
} else {
., err = merge(., structured)
.error_output = "Ok"
}
'''

[sinks.logs_to_aggregator]
type = "vector" # required
inputs = ["empty_remap","filter_notempty"] # required
address = "vector-aggregator:9001" # required

the above doesn't seem to work, we are still getting an empty value in error_output field instead of 'Ok'm any lead would help, thanks.

[tobz]

This boils down to you trying to filter on fields in the undecoded message before actually decoding it. The events from nginx_source_metrics are passing the condition in filter_notempty because technically a missing field does, in fact, not equal an empty string, as it is null.

If you remove the filter transforms, and just tweak the remap slightly, this is achievable:

toby@consigliere:~/testing/empty-field-filtering-13832$ cat input-empty
{"hostname":"xxxxxxx","method":"POST","path":"YYYYYYY","status":"200","request_time":"0.000","partner":"111","cache_status":"HIT","nginx_host":"knginx-785bf9c75f-5mmbq","error_output":""}

toby@consigliere:~/testing/empty-field-filtering-13832$ cat input-notempty
{"hostname":"xxxxxxx","method":"POST","path":"YYYYYYY","status":"200","request_time":"0.000","partner":"111","cache_status":"HIT","nginx_host":"knginx-785bf9c75f-5mmbq","error_output":"Bad"}

toby@consigliere:~/testing/empty-field-filtering-13832$ cat input-notpresent 
{"hostname":"xxxxxxx","method":"POST","path":"YYYYYYY","status":"200","request_time":"0.000","partner":"111","cache_status":"HIT","nginx_host":"knginx-785bf9c75f-5mmbq"}

toby@consigliere:~/testing/empty-field-filtering-13832$ cat vector.toml
data_dir = "/tmp"

[sources.nginx_source_metrics]
type = "stdin"

[transforms.empty_remap]
type = "remap" # required
inputs = ["nginx_source_metrics"] # required
source = '''
structured = parse_json!(del(.message))
. = merge!(., structured)

if is_nullish(.error_output) {
.error_output = "Ok"
}
'''

[sinks.logs_to_aggregator]
type = "console"
inputs = ["empty_remap"] # required
encoding.codec = "json"

toby@consigliere:~/testing/empty-field-filtering-13832$ cat input-empty | ./vector --config vector.toml 
...
{"cache_status":"HIT","error_output":"Ok","host":"consigliere","hostname":"xxxxxxx","method":"POST","nginx_host":"knginx-785bf9c75f-5mmbq","partner":"111","path":"YYYYYYY","request_time":"0.000","source_type":"stdin","status":"200","timestamp":"2022-08-03T20:07:01.099681968Z"}

toby@consigliere:~/testing/empty-field-filtering-13832$ cat input-notempty | ./vector --config vector.toml 
...
{"cache_status":"HIT","error_output":"Bad","host":"consigliere","hostname":"xxxxxxx","method":"POST","nginx_host":"knginx-785bf9c75f-5mmbq","partner":"111","path":"YYYYYYY","request_time":"0.000","source_type":"stdin","status":"200","timestamp":"2022-08-03T20:07:05.700485104Z"}

toby@consigliere:~/testing/empty-field-filtering-13832$ cat input-notpresent | ./vector --config vector.toml 
...
{"cache_status":"HIT","error_output":"Ok","host":"consigliere","hostname":"xxxxxxx","method":"POST","nginx_host":"knginx-785bf9c75f-5mmbq","partner":"111","path":"YYYYYYY","request_time":"0.000","source_type":"stdin","status":"200","timestamp":"2022-08-03T20:07:12.258469998Z"}

Here I've used is_nullish, which will return true in both the case of the value being a true empty string, or if it doesn't exist, in which case it would actually be null.

[dannygueta]

Thanks @tobz , this really helped.