Description
Describe the bug
emit_invalid_record_to_error
in the parser filter is a setting which allows users to decide on the outcome that happens when a record is unable to parsed for one of three reasons.
If set to true it says that the record will be given the @ERROR
label, and "If you want to ignore these errors, set false".
However, I was seeing different behaviour, and looking at the relevant piece of source code https://github.com/fluent/fluentd/blob/c48ca04a593f2f4628c545d74c90f6caba513bee/lib/fluent/plugin/filter_parser.rb#L72-L95
it became clear that the documentation isn't quite clear.
Notably, the source code clearly has an extra condition that if reserve_data
is true then it will not discard these errors (as I would of expected), but instead it allows them to pass but with an unparsed record. I fully understand why this behaviour exists, but I think the documentation around this could be included.
Link to the problematic documentation
https://docs.fluentd.org/filter/parser#emit_invalid_record_to_error
https://docs.fluentd.org/filter/parser#reserve_data
Expected explanation
I would expect the documentation for emit_invalid_record_to_error
to clearly say that if reserve_data
is set to true, then the record will not be discarded but will instead be passed through without any parsing happening.
Additional context
An example configuration which was not clear from the documentation is:
<filter gearset.*>
@type parser
<parse>
@type json
json_parser json
</parse>
key_name log
reserve_data true
emit_invalid_record_to_error false
</filter>
I am happy to open a PR myself, but given this is my first open-source contribution I figured I would put an issue in.