Support for AWS Profiles != "default"

[dougheitkamp]

It looks like manticore_adapter.rb on lines 43 and 44 pull AWS credentials using options that come from http_client.rb (lines 262-293). Nowhere in here can the AWS_PROFILE be set to anything other than "default". I need to be able to specify which AWS profile should be used. My particular use case uses the same profile for all amazon_es outputs. Ideally the profile should be:
1.) the value of a new option, aws_profile, if provided.
2.) the AWS_PROFILE environment variable value, if set and (1) is not provided.
3.) "default", if (1) and (2) are not available.

If nothing else, getting option 2 (and 3) to work makes this plugin consistent with the SQS input plugin.

BTW, the error for this is pretty poor. You get this error...

undefined method `credentials' for nil:NilClass

... which look exactly like #126

It took a long time to figure out what was really happening here. If there is a way to provide the profile and I missed it, please let me know. My ruby is not good, so I'd rather not attempt the fix on this.

[AustinTag]

You can provide static credentials to the plugin, does this not work for your use case?

[dougheitkamp]

Our credentials expire every hour, so while it's possible to use "static" credentials, using this approach is a horrible experience.

Our development teams have access to multiple AWS accounts and we keep simultaneous credentials active. The AWS command line arguments and the AWS programming APIs can all accommodate this multi-account setup by allowing us to set the AWS_PROFILE environment variable or by providing the profile directly (via options like --profile on the aws command line). The SQS input plugin for Logstash is also sensitive the AWS_PROFILE setting. The amazon_es output plugin is not.

Our work-around for this has been to have our devs label the account that they need to do amazon_es work in as "default" in their credentials file. We really frown upon doing this though as it becomes very easy to execute the wrong aws command in the wrong environment when "default" is set. When it's not set, commands will error out, forcing the developer to correct the mistake and explicitly set the intended profile.

More background:
Our production and staging environments don't have this issue, as they are deployed Fargate tasks, but all of our development happens on local docker containers. It's during this development that we have our issues.

[AustinTag]

I see. This feature makes sense. We are prioritizing a few bug fixes before adding new features, but we will update this issue when we have bandwidth for this task.

[dougheitkamp]

Thank you.