logstash failing with bulk load to aws_es issue

[userakhila]

Logstash is failing to upload bulk document to amazon elasticsearch and keeps trying to upload that document infinitely without executing other logs. Is there a way to reject bulk message and process the other ones?

Error:

[ERROR][logstash.outputs.elasticsearch][main] Encountered a retryable error. Will Retry with exponential backoff  {:code=>413, :url=>"https://aws_es_domain:443/_bulk", :body=>"{\"Message\":\"Request size exceeded 10485760 bytes\"}"}

[AustinTag]

This seems related to #114 and #112

[hendricjabs]

I had the same error and I could fix this by removing the protocol in front of the hosts URL.
So instead of

output {
    amazon_es {
        hosts => ["https://foo.eu-central-1.es.amazonaws.com"]
        region => "eu-central-1"
        index => "production-logs-%{+YYYY.MM.dd}"
    }
}

I changed it to the following

output {
    amazon_es {
        hosts => ["foo.eu-central-1.es.amazonaws.com"]
        region => "eu-central-1"
        index => "production-logs-%{+YYYY.MM.dd}"
    }
}

Maybe it would be an idea to check the configuration and throw an error before processing it or to automatically remove the protocol.

[AdrienBigot]

Same problem even using max_bulk_bytes with a value of 100.000 .
Very big problem since it break totaly the ingestion of our logs.
Thanks in advance for a fix 👍

Adrien