kafka-stack-docker-compose

This replicates as well as possible real deployment configurations, where you have your zookeeper servers and kafka servers actually all distinct from each other. This solves all the networking hurdles that comes with Docker and docker-compose, and is compatible cross platform.

Stack version

• Zookeeper version: 3.4.9
• Kafka version: 2.5.0 (Confluent 5.5.1)

Requirements

Docker

Please export your environment before starting the stack:

export DOCKER_HOST_IP=127.0.0.1

(that's the default value and you actually don't need to do a thing)

Docker-Toolbox

If you are using Docker for Mac <= 1.11, or Docker Toolbox for Windows (your docker machine IP is usually 192.168.99.100)

Please export your environment before starting the stack:

export DOCKER_HOST_IP=192.168.99.100

Single Zookeeper / Single Kafka

This configuration fits most development requirements.

• Zookeeper will be available at $DOCKER_HOST_IP:2181
• Kafka will be available at $DOCKER_HOST_IP:9092

Run with:

docker-compose -f zk-single-kafka-single.yml up
docker-compose -f zk-single-kafka-single.yml down

Single Zookeeper / Multiple Kafka

If you want to have three brokers and experiment with kafka replication / fault-tolerance.

• Zookeeper will be available at $DOCKER_HOST_IP:2181
• Kafka will be available at $DOCKER_HOST_IP:9092,$DOCKER_HOST_IP:9093,$DOCKER_HOST_IP:9094

Run with:

docker-compose -f zk-single-kafka-multiple.yml up
docker-compose -f zk-single-kafka-multiple.yml down

Multiple Zookeeper / Single Kafka

If you want to have three zookeeper nodes and experiment with zookeeper fault-tolerance.

• Zookeeper will be available at $DOCKER_HOST_IP:2181,$DOCKER_HOST_IP:2182,$DOCKER_HOST_IP:2183
• Kafka will be available at $DOCKER_HOST_IP:9092

Run with:

docker-compose -f zk-multiple-kafka-single.yml up
docker-compose -f zk-multiple-kafka-single.yml down

Multiple Zookeeper / Multiple Kafka

If you want to have three zookeeper nodes and three kafka brokers to experiment with production setup.

• Zookeeper will be available at $DOCKER_HOST_IP:2181,$DOCKER_HOST_IP:2182,$DOCKER_HOST_IP:2183
• Kafka will be available at $DOCKER_HOST_IP:9092,$DOCKER_HOST_IP:9093,$DOCKER_HOST_IP:9094

Run with:

docker-compose -f zk-multiple-kafka-multiple.yml up
docker-compose -f zk-multiple-kafka-multiple.yml down