Change Data Capture with Debezium, MongoDB, Kafka, and OpenSearch

A robust solution for real-time data synchronization between MongoDB and OpenSearch using Kafka and Debezium.

System Architecture

Prerequisites

Before you begin, make sure the following tools are installed:

• Docker & Docker Compose
• Git
• Basic understanding of MongoDB, Kafka, and OpenSearch

Installation

Clone this repository

git clone https://github.com/stfuxbm/cdc-mongodb.git
cd cdc-mongodb

Modify the MongoDB source connector

config/debezium/mongodb-conn.json

Modify the OpenSearch sink connector

config/debezium/opensearch-sink.json

Run the services using Docker Compose

  docker compose docker-compose.yml

Run Locally

Make the setup script executable and run it

chmod +x scripts/setup-kafka-connect.sh
bash scripts/setup-kafka-connect.sh

Monitoring Data Changes

Make the setup script executable and run it

chmod +x scripts/monitor-kafka-changes.sh
bash scripts/monitor-kafka-changes.sh

Logs

Log files are generated during runtime, including connector logs, errors, and debugging information. These logs are useful for monitoring and troubleshooting purposes.

logs

References

• Debezium Documentation
• Apache Kafka Documentation
• OpenSearch Documentation
• Aiven Documentation