Fix the limitation of the size of yaml file that exceeds 3MB

[kaisecheng]

Fixed: #96

added setting yaml_dictionary_code_point_limit to config the maximum code point limit of the yaml file in dictionary_path to overcome the 3MB size limit from SnakeYaml 1.33. This setting is only effective for yaml. Yaml file over the size limit throws an exception. JSON and CSV currently do not have such restriction. The default value of yaml_dictionary_code_point_limit is 128MB.

How to test

• checkout the branch
• build the gem
  • bundle install
  • gem build logstash-filter-translate.gemspec
• install the gem in Logstash
  • bin/logstash-plugin install --local ~/path/to/your/gem/logstash-filter-translate-3.4.1.gem
• download a 4MB yaml from here
• run the following config

input {
  file {
    path => "/path/to/yaml/big_yaml_4mb.yml"
    start_position => "beginning"
  }
}
filter {
  dissect {
    mapping => {
      "message" => "%{key}: %{val}"
    }
  }
  translate {
    source => "key"
    target => "translated"
    fallback => "UNKNOWN"
    dictionary_path => "/path/to/yaml/big_yaml_4mb.yml"
    yaml_dictionary_code_point_limit => 5000000
  }
}
output {
    stdout { codec => dots }
}

• Logstash should start without error

[jsvd]

Given that there didn't use to be a limit, and I don't think disabling limit checking it is an option (or is it?), I'm tempted to put a very high number here, like 128MB or even 1GB

[kaisecheng]

There is no option to disable the limit check. I put it 128MB

[mashhurs]

Changes overall LGTM by agreeing @jsvd 's comments.
One thing I would like to clarify here, Psych moved from snakeyaml to snakeyaml-engine this year. The reason behind as I understood to mitigate the CVE risks and performances (snakeyaml-engine is more suitable in JVM env).
This change also indicates that under the hood we are moving to snakeyaml-engine AND for the long term we should/may move our snakeyaml dependencies to snakeyaml-engine as well.

Reference thread: link

[mashhurs]

👍

[jsvd]

A few leftovers from the max_bytes setting.
Also, I tried locally with refresh_interval and increasing the dictionary over the code point limit, and the reloader thread crashed:

[2023-05-11T23:26:31,445][ERROR][logstash.filters.translate][main] Scheduler intercepted an error: {:exception=>NoMethodError, :message=>"private method `warn' called for nil:NilClass", :backtrace=>["/Users/joaoduarte/elastic/logstash-plugins/logstash-filter-translate/lib/logstash/filters/dictionary/file.rb:123:in `loading_exception'", "/Users/joaoduarte/elastic/logstash-plugins/logstash-filter-translate/lib/logstash/filters/dictionary/file.rb:60:in `load_dictionary'",

The two @logger.warn in file.rb need to be changed to just logger.warn to produce the right error message:

[2023-05-11T23:22:42,793][WARN ][logstash.filters.dictionary.yamlfile][main] Translate: The incoming YAML document exceeds the limit: 100 code points. when loading dictionary file at /tmp/dict.yml, continuing with old dictionary {:dictionary_path=>"/tmp/dict.yml"}

[jsvd]

I left a suggestion to avoid leaking the yaml code point limit setting to the base dictionary file initialize and initialize_for_file_type method signatures, by using an optional argument hash.

[jsvd]

LGTM