Skip to content

Releases: ntnhaatj/cdc-ingestion-debezium

v0.1 SQL Table CDC

18 May 02:21
@ntnhaatj ntnhaatj
v0.1
fc77ab9
Compare
Choose a tag to compare
v0.1 SQL Table CDC Latest
Latest

What's Changed

  • One docker-compose command to launch the POC demo.
  • Ability to capture all changes in all source tables.
  • This demo version uses the same SQL instance for sink tables.

POC demo images

  • Tables with suffix _cdc is for capturing data changes on each source respectively

Screen Shot 2022-05-18 at 09 27 15

  • For instance customer table schema

Screen Shot 2022-05-18 at 09 27 40

  • CDC schema of customer table

Screen Shot 2022-05-18 at 09 27 34

Notes:

  • op field captured the action on source table: u -> update, c -> create, d -> update

Limitation

  • Hasn't covered the feature of joining source tables on every change.
  • Hasn't written script for stress test as well as service for checking data quality (DBT).

Full Changelog: https://github.com/ntnhaatj/cdc-ingestion-debezium/commits/v0.1

Upcoming

  • Join all related source tables on changes
  • Stress test and data quality on sink tables
Assets 2
Loading