Action/metadata line ... contains an unknown parameter [_retry_on_conflict]

[megahall]

I am hitting the following bug, which has been reported before with the standard ES output plugin, inside of the AWS ES output plugin:

logstash-plugins/logstash-output-elasticsearch#710
logstash-plugins/logstash-output-elasticsearch#812

It appears when you are trying to operate the plugin against recent ES releases, including AWS ES releases, with the update / upsert based workflow.

The following one-line change fixes it:

--- common.rb      2019-11-26 01:40:47.590958913 +0000
+++ logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/common.rb      2019-11-26 01:33:58.900352556 +0000
@@ -88,7 +88,7 @@
       if action == 'update'
         params[:_upsert] = LogStash::Json.load(event.sprintf(@upsert)) if @upsert != ""
         params[:_script] = event.sprintf(@script) if @script != ""
-        params[:_retry_on_conflict] = @retry_on_conflict
+        params[:retry_on_conflict] = @retry_on_conflict
       end

       if @version

However, the logic inside this plugin is somewhat different than the other plugin, which determines the flag naming based on the ES code version, so either this would need to be a blanket fix, or somebody more expert on this version of the could would need to help understand how to name the flag more dynamically.

When this issue occurs, a mysterious error is raised in logstash which causes it to retry requests forever and ever, but always failing:

Action/metadata line ... contains an unknown parameter [_retry_on_conflict]

[dougheitkamp]

Any update on this bug?

[bkstorm]

I think I have the same issue, but with _routing

Action/metadata line [1] contains an unknown parameter [_routing]

[rrrongon]

I am trying to make bulk update request with painless. While, I try to update without _routing, AWS Elasticsearch responses with Missing routing param Exception. On the other hand, While I add _routing and try it with Elasticsearch Client with python for v.7x it rejects the request with RequestError(400, illegal_argument_exception, Action/metadata line [1] contains an unknown parameter [_routing]) means client is not aware of this param. Seems they are conflicting each other.

[abdulsamad19951]

Still facing the same issue reported by @megahall.
On update and upsert bulk upload
Receiving following error

Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"https://**********/_bulk", :body=>"{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_retry_on_conflict]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_retry_on_conflict]\"},\"status\":400}"}

[rishinaveen]

Is making the fix suggested by @megahall and building the plugin locally, the only way to get this working?

[fpalluel]

Hello,
We are bitten by this bug too : cannot upsert any document.
Is it possible to make a new release implementing the @megahall small fix ?
Are you waiting for a PR ?

Thanks.

[theaquarian007]

Any update on this one. I am also running into the same issue but with _routing

[AndrewChau]

It's hard to have confidence in using this plugin in production when we have to rely on the "blanket fix" from the OP.
Upsert is a critical part for us so we'll have to re-evaluate our decision to use Amazon's Elasticsearch service.