Send Docker container's logs to Logstash

[jmarin]

The logback-logstash integration is JVM specific (i.e. won't work for other tech) and has been somewhat flaky at times. It also uses TCP which might affect performance of the running application. Explore ways to send logs from running Docker container to Logstash. At the same time, review codebase to standardize and increase logging

[hkeeler]

Renaming this to "Send Docker container's logs to Logstash". I think the review codebase to standardize and increase logging is covered by #105.

[hkeeler]

I've been working though Nathan LeClaire's Automating Docker Logging: ElasticSearch, Logstash, Kibana, and Logspout post. Here's the TL;DR:

# Upgrade docker-compose to 1.2.0.  Necessary to support `extends` in compose yaml files
brew update
brew upgrade docker-compose

# Checkout his canned ELK/Logspout setup
git clone https://github.com/nathanleclaire/elk nathanleclaire-elk

# Launch it
cd nathanleclaire-elk
docker-compose up

...then browse to the following:

Service	URL
Kibana	http://{{boot2docker-ip}}
Elasticsearch	http://{{boot2docker-ip}}:9200
cAdvisor	http://{{boot2docker-ip}}:8080

I'll be continuing to work through this post, and then try to merge a hopefully simpler version into our Docker Compose setup.

[hkeeler]

I've been reading through Centralized System and Docker Logging with ELK Stack, which takes the approach we've been discussed:

1. Run an rsyslog service...somewhere.
2. Run each Docker containers with --log-driver syslog, sending logs to syslog instead of stdout.
3. rsyslog forwards to Logstash.
4. Logstash writes to Elasticsearch

The piece of this puzzle I didn't completely understand is that the rsyslog service is on the Docker host, not another container...which makes sense. However, this now means that if you're running Logstash as a Docker container, the host now has a dependency on a container. This seems problematic for dynamic environments. How does the host know where Logstash is? Standard DNS name per env? Use docker-gen?

In addition, this makes for a strange startup order for new environments:

1. Create new Docker host with rsyslog service running.
2. Create Logstash Docker container.
3. Update hosts's rsyslog config with Logstash's IP/name...and restart.

This seems awkward, but maybe there's a solution again with docker-gen.

[hkeeler]

I think I'm leaning back to the STDOUT -> logspout -> Logstash, even with its limitations. See #104 (comment).