Credential Error on Shutdown

[LogicWolfe]

After the logstash process receives a TERM signal I start seeing errors:

[2018-11-30T22:48:36,894][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<NoMethodError: undefined method `credentials' for nil:NilClass>}
[2018-11-30T22:48:36,895][ERROR][logstash.outputs.elasticsearch] An unknown error occurred sending a bulk request to Elasticsearch. We will retry indefinitely {:error_message=>"undefined method `credentials' for nil:NilClass", :error_class=>"NoMethodError", :backtrace=>["/srv/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.136/lib/aws-sdk-core/signers/v4.rb:45:in `initialize'", "/srv/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.1.pre-java/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb:111:in `perform_request'", "/srv/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.1.pre-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:291:in `perform_request_to_url'", "/srv/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.1.pre-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:278:in `block in perform_request'", "/srv/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.1.pre-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:373:in `with_connection'", "/srv/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.1.pre-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:277:in `perform_request'", "/srv/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.1.pre-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:285:in `block in Pool'", "/srv/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.1.pre-java/lib/logstash/outputs/amazon_es/http_client.rb:145:in `bulk_send'", "/srv/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.1.pre-java/lib/logstash/outputs/amazon_es/http_client.rb:130:in `bulk'", "/srv/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.1.pre-java/lib/logstash/outputs/amazon_es/common.rb:275:in `safe_bulk'", "/srv/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.1.pre-java/lib/logstash/outputs/amazon_es/common.rb:180:in `submit'", "/srv/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.1.pre-java/lib/logstash/outputs/amazon_es/common.rb:148:in `retrying_submit'", "/srv/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.1.pre-java/lib/logstash/outputs/amazon_es/common.rb:38:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:114:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97:in `multi_receive'", "/srv/logstash/logstash-core/lib/logstash/pipeline.rb:373:in `block in output_batch'", "org/jruby/RubyHash.java:1343:in `each'", "/srv/logstash/logstash-core/lib/logstash/pipeline.rb:372:in `output_batch'", "/srv/logstash/logstash-core/lib/logstash/pipeline.rb:324:in `worker_loop'", "/srv/logstash/logstash-core/lib/logstash/pipeline.rb:286:in `block in start_workers'"]}

These continue indefinitely preventing a clean shutdown. This may be related to threading issued with aws-sdk v2: aws/aws-sdk-ruby#1857.

This is with Logstash 6.4.2 and a gem built from master in this repo.

[crisidev]

This looks to be related to the issue you are referencing in the aws-sdk.. We will need to investigate this.

Logstash supports a flag to force shutdown the pipeline no matter what: https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html

pipeline.unsafe_shutdown | When set to true, forces Logstash to exit during shutdown even if there are still inflight events in memory. By default, Logstash will refuse to quit until all received events have been pushed to the outputs. Enabling this option can lead to data loss during shutdown.

I am not sure if it fits your usecase and it can lead to some data loss, but it can be a workaround for the time being.