Add AWS Elasticsearch image

[AdrienKuhn]

This PR adds debian-awselasticsearch docker image.

AWS ES service require requests signature.

This image contains the aws-elasticsearch-service fluent plugin.

Authentication with AWS ES is done using EC2 instance profile.

[Jaykah]

Is anyone using this in production?

Also, any plans to merge?

Thanks!

[repeatedly]

Currently, no plan to merge because we have a problem to maintain contributed images.

#293

[danopia]

I've been using an older version of the same thing in production for at least a year now. Just switched my production to this image because I needed a newer elasticsearch plugin. Dropped it right in 👍

I also replace /fluentd/etc with my own configuration at the Kubernetes level, so that I can roll out new configs without making new images. Works well, but sadly means I can't directly vouch for the config presented here.

[gatolim]

@AdrienKuhn thanks for coming up with the repo. I am new to the docker world and was playing around this. I have my ES set up in AWS and have docker for window set up locally. I cloned your repo and docker build the debian-awselasticsearch image however when I try to apply the image using fluentd-daemonset-awselasticsearch-rbac.yml I have the following error:
[FATAL tini (6)] exec /fluentd/entrypoint.sh failed: No such file or directory

I verified that the build steps does copy this .sh file and the file existed in my build folder. Am I missing something ?

[AdrienKuhn]

@gatolim I tried to apply the fluentd-daemonset-awselasticsearch-rbac.yml file on my side and I didn't encounter any issue.

I used the following (v1.7) image for test purposes: krewh/fluentd-awselasticsearch.
We are also using the 1.4 version (deployed through a custom Helm chart) in a dev cluster.

[gatolim]

Thanks @AdrienKuhn, I ended up building my own image. I encountered the same problem while using my own and it turns out to be the end line char ('\r\n') difference between window and linux in the .sh file. I suspected it was the same cause when I use yours.

[wcarlsen]

We ended up using this proxy (https://github.com/abutaha/aws-es-proxy) as a sidecar, which allows us to use on of the pre built images of this repos. Works flawlessly, but the proxy image development is rather stale.

[Nuru]

We ended up using this proxy (https://github.com/abutaha/aws-es-proxy) as a sidecar, which allows us to use on of the pre built images of this repos. Works flawlessly, but the proxy image development is rather stale.

@wcarlsen Did you really get aws-es-proxy to work flawlessly? Last time I tried it I had to reject it because if it encountered a 403 or 302 status code from Elasticsearch it would just swallow it and return an empty 200 to the caller. How did you fix this, or did you just decide it was acceptable behavior?

[wcarlsen]

@Nuru we probably shouldn't continue this here (hint: email me). But I wasn't aware of this behavior. I think it is a trade off between maintaining a fluentd image. We are currently investigating when we would expect to get a 302 or 403 from AWS ES. Maybe you have experience with this or have a better overview of how big a problem this actually is?