Skip to content

Unclear description of logic that happens on an error in the "parser filter" #442

Open
@jwreford99

Description

@jwreford99

Describe the bug

emit_invalid_record_to_error in the parser filter is a setting which allows users to decide on the outcome that happens when a record is unable to parsed for one of three reasons.
If set to true it says that the record will be given the @ERROR label, and "If you want to ignore these errors, set false".

However, I was seeing different behaviour, and looking at the relevant piece of source code https://github.com/fluent/fluentd/blob/c48ca04a593f2f4628c545d74c90f6caba513bee/lib/fluent/plugin/filter_parser.rb#L72-L95
it became clear that the documentation isn't quite clear.

Notably, the source code clearly has an extra condition that if reserve_data is true then it will not discard these errors (as I would of expected), but instead it allows them to pass but with an unparsed record. I fully understand why this behaviour exists, but I think the documentation around this could be included.

Link to the problematic documentation

https://docs.fluentd.org/filter/parser#emit_invalid_record_to_error
https://docs.fluentd.org/filter/parser#reserve_data

Expected explanation

I would expect the documentation for emit_invalid_record_to_error to clearly say that if reserve_data is set to true, then the record will not be discarded but will instead be passed through without any parsing happening.

Additional context

An example configuration which was not clear from the documentation is:

<filter gearset.*>
  @type parser
  <parse>
    @type json
    json_parser json
  </parse>
  key_name log
  reserve_data true
  emit_invalid_record_to_error false
</filter>

I am happy to open a PR myself, but given this is my first open-source contribution I figured I would put an issue in.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions