Releases: apache/flink-cdc
Releases · apache/flink-cdc
Release-2.2.1
Release Notes
Improvements and Bug fixes
- [docs] Update the dependencies version used in docs to 2.2.0
- [hotfix][docs] Fix jar link for PolarDB-X docs (#1013)
- [docs] Add example for Scan Newly Added tables feature (#1016)
- [hotfix][docs] Fix misspell of docs title
- [docs][mysql] Add note to make the tableList usage and dependencies clearer (#1037)
- [docs] Update the supported sink vendors
- [sqlserver] Add more supported(tested) SqlServer versions (#1049)
- [oceanbase] Remove useless dependency and code of test (#1046)
- [docs][mysql] Fix connector document link error (#1043)
- [mysql] Serialize isSuspended flag for MySqlBinlogSplit (#1055)
- [common] Apply filter on ResolvedSchema to keep physical columns only (#1097)
- [docs][mongodb] Update document about MongoDB database and collection regex filter (#1090)
- [mysql] Throw RuntimeException timely in snapshot scan phase (#1099)
- [hotfix][tidb] Fix RowKind.INSERT judgement condition (#1104)
- [mysql] Catch and hold exception in chunk splitter thread and rethrow in getNext() (#1110)
- [mysql] Use the gtid set firstly when compare the binlog offset (#1111)
- [mysql] Add heartbeatInterval field to equals() and hashCode() (#1118)
Download
- flink-sql-connector-mongodb-cdc-2.2.1.jar
- flink-sql-connector-mysql-cdc-2.2.1.jar
- flink-sql-connector-oceanbase-cdc-2.2.1.jar
- flink-sql-connector-oracle-cdc-2.2.1.jar
- flink-sql-connector-postgres-cdc-2.2.1.jar
- flink-sql-connector-sqlserver-cdc-2.2.1.jar
- flink-sql-connector-tidb-cdc-2.2.1.jar
Documentation
Release-2.2.0
Release Notes
Documentation: https://ververica.github.io/flink-cdc-connectors/release-2.2/
Features and Improvements
- [docs] Update the supported database vendors and improve the notes.
- [docs][mysql][polardbx] Add polardbx tutorials (#1003)
- [docs][mysql] Remove redundant docker environment variable in tutorials (#1005)
- [cdc-base] Add MySQL Incremental Source to validate the Incremental Snapshot Framework(#776)
- [cdc-base] Improve the Incremental Snapshot Interfaces
- [cdc-base] The first version of Incremental Snapshot Framework abstraction
- [hotfix][docs] Fix markdown title due to typo in readme
- [oceanbase] Bump the oblogclient version to 1.0.3 to fix the NPE in oblogclient connection (#998)
- [docs] Clarify ownership to comply with ASF trademark policies (#1000)
- [build] Decrease the azure job timeout to reasonable 40 minutes (#991)
- [hotfix][site] Use Flink with its trademark
- [docs] Adopt the full name 'Apache Flink' with its trademark notice (#988)
- [hotfix][docs] Update the copyright
- [docs][mysql] Correct the jdbc driver version in documentation
- [docs] Organize the connector docs in alphabetical order
- [docs][tidb] Add tidb cdc connector document
- [hotfix][docs] Fix typo in tidb-tutorial.md and tidb-tutorial-zh.md
- [mysql] Suspend SourceReader eagerly to reduce restoration time
- [docs][oceanbase] Add quickstart tutorials for OceanBase CDC connector
- [docs][oceanbase] Add docs for OceanBase CDC connector
- [build][oceanbase] Use self-hosted agent for oceanbase cdc tests
- [oceanbase] Add OceanBase CDC connector
- [tidb] remove useless code.
- [test][e2e] Use single fork in tests
- [test][e2e] Clean up database resources for every e2e tests.
- [e2e] bump flink e2e test version.
- [tidb] Add tidb cdc connector e2e test.
- [tidb] Add flink1.14 support.
- [hotfix][mysql] Avoid schema isn't known error when scanNewlyAddedTable enabled (#965)
- [mysql][polardbx] Remove debug print code in PolardbxSourceTCase (#968)
- [tidb] Introduce runtime converter to improve the deserialization
- [tidb][test] Add timezone tests for tidb
- [tidb] Remove useless config options
- [docs][site] Add favicon fot website index html
- [docs][site] Introduce static index html for the project website (#961)
- [mysql] Split snapshot chunks asynchronously (#931)
- [docs][tidb] Add index for tidb tutorial docs
- [tidb][docs] Add tidb connector tutorial (#938)
- [tidb] Add metadata column ITCase and improve the code style
- [tidb][test] Fix tikv port conflicts
- [tidb] Support metadata column
- [tidb][test] Add tidb pipeline in azure CI
- [tidb][test] Add TiDBTableSourceFactoryTest and TiDBConnectorITCase
- [tidb] Fix tikv.pd.addresses not work and use ResolvedSchema instead of TableSchema.
- [tidb] Bump tikv client version to v3.2.0
- [tidb] The first draft of tidb cdc connector
- [docs] List supported databases in docs (#954)
- [mysql] Fallback to build schema by 'desc table' case when parsing DDL failure (#949)
- [docs] Fix download url in mongodb tutorial (#951)
- [docs] Update DingTalk QR code to English version (#950)
- [mongodb] Match multiple database and collection names using a regular expression in MongoDB. (#940)
- [mysql] Allow custom JDBC URL parameters in MySQL CDC source (#921)
- [mongodb][test] Fix unstable tests of MongoDBConnectorITCase (#944)
- [mysql] Improve mysql heartbeat feature and tests
- [mysql] Original implementation of mysql heartbeat feature
- [common] Update issue templates
- [hotfix][mysql] Fix MySqlSourceReader#snapshotState should return unfinishedSplits (#926)
- [docs] Fix the outdated link of flink download link (#922)
- [hotfix][mysql] Fix the binlog position didn't update when emit DDL (#925)
- [mongodb] Correct the TypeInformation argument of DebeziumDeserializationSchema in SourceFuntion (#780)
- [common] Allow user to pass custom JsonConverter configs in JsonDebeziumDeserializationSchema (#615)
- [mysql] Improve the test to cover table with varbinary primary key (#879)
- [mysql] Support primary key with varbinary type (#879)
- [format][changelog-json] Remove the support of changelog-json format (#912)
- [hotfix][minor] Fix the test code typo
- [mysql] Fix ArrayIndexOutOfBoundsException in mysql binlog read phase (#911)
- [mongodb][tests] Track MongoDBITCase hang in azure (#881)
- [docs] Update the supported flink version mapping (#885)
- [common] Fix emitDelay and fetchDelay metrics to be 0L during snapshot phase (#878)
- [hotfix][docs][mongodb] Update supported mongodb versions in README.md
- [docs][sql-server] Add sqlserver-cdc connector tutorial (#687) This closes #687.
- [common] Use ResolvedSchema instead of deprecated TableSchema API (#782)
- [docs][sqlserver] Add SqlServer CDC connector document (#705)
- [doc][sqlserver] Add SqlServer English tutorial document. (#693)
- [common] Make Flink CDC Compatible with Flink 1.14
- [build] Bump flink version to 1.13.5
- [docs][oracle] Add guidance about oracle CDB database (#792)
- [docs] Add connection.pool.size and connect.max-retries option in mysql document (#857)
- [sqlserver] Add e2e tests for sqlserver cdc connector (#837)
- [hotfix][mysql] Rename method from dispatchHighWatermark to dispatchBinlogEndEvent (#810) This closes #810.
- [mysql] Use READ type for all records in snapshot reading (#842)
- [mysql] Rewrite the newly added table tests to cover more cases (#777) This closes #777.
- [mysql] Improve newly added tables process
- [mysql] Support capture newly added tables for existed pipeline
- [docs] Fix typo in mongodb-cdc.md (#836)
- [mysql] add test to cover the datatype GeomCollection that's synonymous with GeometryCollection for mysql8 (#822)
- [sqlserver] Add SqlServer cdc connector (#662)
- [build] Remove redundant maven enforce plugin (#794)
- [hotfix] Fix MySqlSource comment typo (#793)
- [mysql] Support MySQL 5.6 (#396)
- [build] Switch to Log4j 2 by default (#791)
- [oracle] Fix Oracle numeric types conversion (#560)
- [hotfix][mysql] Fix message timestamp of snapshot record to 0L (#752)
- [docs] Translate the Chinese FAQ document to English (#737)
- [docs] Remove the fixed version of website (#730)
- [hotfix][mysql] Set minimum connection pool size to 1 (#692)
- [build] Bump log4j2 version to 2.16.0 (#714)
- [build] Remove override definition of maven-surefire-plugin in connectors pom (#673)
- [docs] Add FAQ for tableList option usage in DataStream API (#634)
- [mongodb] Support to connect MongoDB without authentication (#642)
- [hotfix] Fix the parameter typo in java doc (#651)
- [mysql] Add a method to get MySqlSourceConfigFactory from MySqlSource (#648)
- [mysql] Set default driver class name for PooledDataSource (#659)
- [mysql] Fix there is duplicate records outputted when keep updating table during snapshot read (#592)
- [docs] Improve column name note in oracle cdc document. (#625)
- [docs] Add Flink cdc eco-system picture (#623)
- [hotfix][docs] Fix typo in oracle-cdc.md (#613)
- [docs] Add supported Flink versions for Flink CDC 2.1(#611)
- [build] Bump version to 2.2-SNAPSHOT
Download
- flink-sql-connector-mongodb-cdc-2.2.0.jar
- flink-sql-connector-mysql-cdc-2.2.0.jar
- flink-sql-connector-oceanbase-cdc-2.2.0.jar
- flink-sql-connector-oracle-cdc-2.2.0.jar
- flink-sql-connector-postgres-cdc-2.2.0.jar
- flink-sql-connector-sqlserver-cdc-2.2.0.jar
- flink-sql-connector-tidb-cdc-2.2.0.jar
Contributor List (sorted alphabetically)
- Dian Qi
- He Wang
- Jark Wu
- Jiabao Sun
- Jingsong Lee
- Junwang Zhao
- Konstantin Knauf
- Leonard Xu
- Mustang Kong
- Paul Zhang
- Qingsheng Ren
- Sergey Nuyanzin
- Shengkai
- Tuple
- Xianxun Ye
- Xuebing Li
- Zongwen Li
- camelusluo
- eastfisher
- ehui
- fangpc
- gongzhongqiang
- hele.kc
- hiscat
- luoyuxia
- paul8263
- skylines
- szgeqin
- taox
- tsreaper
- wangxiaojing
- wysstartgo
- yexianxun
- 元组
Release-2.1.1
Release Notes
Improvements and Bug fixes
- [docs] Remove the fixed version of website (#730)
- [hotfix][mysql] Set minimum connection pool size to 1 (#692)
- [build] Bump log4j2 version to 2.16.0 (#714) Note: This project only uses log4j2 in test code and won't be influenced by log4shell vulnerability
- [build] Remove override definition of maven-surefire-plugin in connectors pom (#673)
- [docs] Add FAQ for tableList option usage in DataStream API (#634)
- [mongodb] Support to connect MongoDB without authentication (#642)
- [hotfix] Fix the parameter typo in java doc (#651)
- [mysql] Set default driver class name for PooledDataSource (#659) (#661)
- [mysql] Fix there is duplicate records outputted when keep updating table during snapshot read (#592)
- [docs] Improve column name note in oracle cdc document (#633)
- [docs] Add Flink cdc eco-system picture (#624)
- [hotfix][docs] Fix typo in oracle-cdc.md(#616)
- [docs] Add supported Flink versions for Flink CDC 2.1(#608)
Download
- flink-sql-connector-mysql-cdc-2.1.1.jar
- flink-sql-connector-postgres-cdc-2.1.1.jar
- flink-sql-connector-oracle-cdc-2.1.1.jar
- flink-sql-connector-mongodb-cdc-2.1.1.jar
- flink-format-changelog-json-2.1.1.jar
Documentation
Release-2.1.0
Release Notes
Documentation: https://ververica.github.io/flink-cdc-connectors/release-2.1/
Features and Improvements
- [tests] Add end-to-end tests for cdc connector uber jars (#594)
- [tests] Upgrade testcontainers dependency to 1.16.2 (#591)
- [mysql] Reading a null column with default value defined should return null instead of default value (#555)
- [mysql] Optimize the dynamic chunk size calculation for table with dense distribution (#599)
- [mongodb] Fix MongoDBConnectorITCase is timeout on Azure (#600)
- [hotfix][mysql] Fix number overflow in ObjectUtils.minus (#598)
- [mongodb] Fix problem of cannot look up fields on non-struct type (#588)
- [hotfix][mongodb] Fix NullPointException when heartbeat option is enabled (#586)
- [postgres] Support 'schema-name' metadata column for postgres-cdc connector
- [oracle] Support metadata columns for oracle connector
- [tests] Split the CI pipeline into multiple stages for better performance (#572)
- [hotfix][mongodb] Correct the connection uri building of MongoDB (#568)
- [mysql] Skip ddl events in snapshot split normalization (#577)
- [mysql] Support dynamic chunk size for evenly distribution table
- [mysql][minor] Use 'table.include.list' key instead of legacy key 'table.whitelist' in SnapshotSplitReader
- [mysql] Set server time zone information for connection pool factory (#566)
- [mysql] Polish binlog offset compare semantic (#543)
- [build] Update NOTICE file for sql connector jar (#565)
- [postgres] Support metadata columns for postgres-cdc connector (#500)
- [debezium] Bump up debezium version to 1.5.4.Final (#554)
- [oracle] Shade dependencies for oracle-cdc connector (#519)
- [hotfix][docs] Fix the display of github star button
- [common] Fix NPE of JsonConverter when job run on cluster(#470)
- [mongodb] Support metadata columns for mongodb-cdc connector (#476)
- [mysql] Reduce the connection usage in table discovery process
- [build] bump maven shade plugin version to 3.2.4
- [mysql] Introduce Jdbc connection pools
- [mysql] support all mysql data types (#438)
- [hotfix][docs] Correct the Kibana link in Oracle connector documentation (#539)
- [mysql] Support binlog split with huge snapshot split meta
- [mysql] Support tables with different split key
- [mysql] Validate the regex pattern of database-name and table-name in table factory (#526)
- [build] Add necessary shaded dependencies (#512)
- [oracle][test] Fix oracle tests failure in azure due to ORA-01882
- [oracle][minor] Fix checkstyle and typo
- [oracle] Add docker build 'on-the-fly' and refactor test image
- [oracle] Added validator for oracle
- [oracle] Updated package naming to com.ververica in oracle connector
- [oracle] Fixing unit tests, cleaning up after adding oracle cdc
- [oracle] Add oracle cdc connector
- [build] Upgrade azure environment from ubuntu-16.04 to ubuntu-18.04 (#517)
- [hotfix][mysql] Correct the usage of includeSchemaChanges parameter (#501)
- [build] Follow Flink builtin connector shade style (#49)
- [mysql] Expose MySqlSource API that supports incremental snapshot (#495)
- [mysql] Support metadata columns for mysql-cdc connector (#496)
- [common] Support to read tables of inconsistent schema (#494)
- [build] Fix maven connection timeout issue in azure CI
- [mysql] Support wide table with huge number of columns
- [tests] Correct the argument order of assertArrayEqualInOrder/assertArrayEqualAnyOrder
- [tests] Introduce assertEqualsInOrder/assertEqualsInAnyOrder methods for test
- [tests] Improve the unstable test testStartupFromLatestOffset in MySqlConnectorITCase
- [mysql] Fix the restoration failure when checkpoint happened in one MySQL transaction
- [mysql] Optimization that skips the binlog read if the low watermark equals high watermark
- [mysql] Add source metrics for MySqlParallelSource
- [build] Use tools with stable version in building docs
- [mysql] Use min/max of split key and row count to describe the distribution of table
- [mysql] Fix the ClassCastException in RecordUtils
- [mongodb][docs] Add documentation for mongodb-cdc connector (#395)
- [mongodb] Remove no need config option prefix 'debezium.' (#420)
- [mongodb] Support MongoDB CDC connector (#225)
- [common] Shade kafka-related dependency (#386)
- [common] Support json deserialization schema
- [mysql] Improve the tests code
- [mysql] Fix the MySqlSnapshotSplitAssigner may assign duplicate splits
- [mysql] Add tests to check mysql-cdc works well under various timezones (#367)
- [mysql] Optimize the checkpoint be optional under single parallelism (#365)
- [postgres] Fix postgres-cdc connector cannot recognize the optional option 'slot.name'
- [mysql] Correct the initial binlog offset for MySqlHybridSplitAssigner
- [mysql] Add default value for 'database.history.instance.name' in MySqlParallelSource
- [common] Remove useless parameter 'converter.schemas.enable'
- [build] Run integration tests for each building
- [mysql] Improve the code format in SignalEventDispatcher
- [postgres] Fix Validator didn't implement Serializable
- [changelog] fix changelog-deserialize exception message typo.
- [build] Update issue templates
- [docs] Add tutorial about building real-time data lake for MySQL sharding table with Flink CDC (#551)
- [docs] Add metadata column example for mysql-cdc and mongodb-cdc connector
- [docs] Add checkpoint configuration limitation for Oracle and Postgres connector (#580)
- [docs] Fix the link of 'edit on github' on site (#578)
- [docs] Use SNAPSHOT version dependency in master documentation (#497)
- [docs] Use more generic term 'transaction logs' in README
- [docs] Add MySQL HA cluster support explanation
- [docs] Update the dependency version to 2.1.0
- [docs] Add checkpoint configuration limitation for Oracle and Postgres connector
- [docs] Fix the link of 'edit on github' on site (#578)
- [docs][oracle] Fix oracle DATE type mapping in documentation (#569)
- [docs] Translate "Streaming ETL for MySQL and Postgres with Flink CDC" into Chinese (#550)
- [docs] Add GitHub star button to documentation (#547)
- [docs] Add Oracle support in README
- [docs] Restructure the "tutorials" to "quickstart"
- [docs] Add documentation for oracle-cdc (#515)
- [docs] Rewrite tutorial doc for mysql-postgres-tutorial (#514)
- [docs] Add favicon for docs website (#525)
- [docs] Use new MySqlSource API in documentation
- [docs] Correct DataStream API usage in document examples
- [docs] Use JsonDeserializationSchema as output format in examples
- [docs] Update documentation of tutorial
- [docs] Add meaningful title for tutorial
- [docs] Support build document website locally
- [docs] Fix the broken link in readme
- [docs] Update the debezium document link to version 1.5
- [docs][mysql] Fix LowerCase Class name typo
- [docs] Add checkpoint config in document example
- [docs] Add FAQ for MySQL 8.0 Public Key Retrieval setting
- [docs] Add checkpoint and primary key setting for example in tutorials
- [docs] Fix the imports in document demo
- [docs] Set the content width of the website
- [docs] Update DingTalk group picture size in README.md
- [docs] Update the supported connector version and link
Download
- flink-sql-connector-mysql-cdc-2.1.0.jar
- flink-sql-connector-postgres-cdc-2.1.0.jar
- flink-sql-connector-oracle-cdc-2.1.0.jar
- flink-sql-connector-mongodb-cdc-2.1.0.jar
- flink-format-changelog-json-2.1.0.jar
Contributor list
Gunnar Morling, Jark Wu, Jiabao Sun, Leonard Xu, MartijnVisser, Marton Balassi, Shengkai, Tamas Kiss, Tamas Zseder, Zongwen Li, dongdongking008, frey66, gongzhongqiang, ili zh, jpatel, lsy, luoyuxia, manmao, mincwang, taox, tuple, wangminchao, yangbin09
Release 2.0.2
Release Notes
Improvements and Bug fixes
- [mysql] Fix the MySqlSnapshotSplitAssigner may assign duplicate splits
- [mysql] Fix the restoration failure when checkpoint happened in one MySQL transaction
- [mysql] Fix the ClassCastException in RecordUtils
- [mysql] Optimization that skips the binlog read if the low watermark equals high watermark
- [mysql] Support wide table with huge number of columns
- [mysql] Use min/max of split key and row count to describe the distribution of table
- [mysql] Add source metrics for MySqlParallelSource
- [tests] Correct the argument order of assertArrayEqualInOrder/assertArrayEqualAnyOrder
- [tests] Introduce assertEqualsInOrder/assertEqualsInAnyOrder methods for test
- [tests] Improve the unstable test testStartupFromLatestOffset in MySqlConnectorITCase
- [docs] Fix the broken link in readme
- [docs] Correct DataStream API usage in document examples
- [docs] Support build document website locally
- [docs] Update documentation of tutorials
- [docs] Add meaningful title for tutorials
- [build] Use tools with stable version in building docs
- [build] Fix maven connection timeout issue in azure CI
Download
- flink-sql-connector-mysql-cdc-2.0.2.jar
- flink-sql-connector-postgres-cdc-2.0.2.jar
- flink-format-changelog-json-2.0.2.jar
Documentation
Release 2.0.1
Release Notes
Improvements and Bug
- [postgres] Fix Validator didn't implement Serializable
- [mysql] Correct the initial binlog offset for MySqlHybridSplitAssigner
- [mysql] Optimize the checkpoint be optional under single parallelism
- [postgres] Fix postgres-cdc connector cannot recognize the optional option 'slot.name'
- [mysql] Improve the code format in SignalEventDispatcher
- [mysql] Add default value for 'database.history.instance.name' in MySqlParallelSource
- [mysql] Add tests to check mysql-cdc works well under various timezones
- [common] Remove useless parameter 'converter.schemas.enable'
- [build] Run integration tests for each building
- [changelog] fix changelog-deserialize exception message typo
- [docs] Add FAQ for MySQL 8.0 Public Key Retrieval setting
- [docs] Update the debezium document link to version 1.5
- [docs] Add checkpoint and primary key setting for example in tutorials
Download
- flink-sql-connector-mysql-cdc-2.0.1.jar
- flink-sql-connector-postgres-cdc-2.0.1.jar
- flink-format-changelog-json-2.0.1.jar
Documentation
Release 2.0.0
Release Notes
Documentation: https://ververica.github.io/flink-cdc-connectors/release-2.0/
Features and Improvements
- [mysql] Support MySQL-CDC 2.0 which offers parallel reading, lock-free and checkpoint feature
- [mysql] Enable single server id for MySqlParallelSource
- [mysql] Keep the global order of between snapshot records and binlog records
- [mysql] Support binlog reader merge optimization
- [mysql] Support numeric type optimization for split boundary calculation
- [mysql] Support table with combined primary key
- [mysql] Support latest-offset mode for MySqlParallelSource
- [build] Rename group.id and package name from 'com.alibaba.ververica' to 'com.ververica'
- [build] Bump flink version to 1.13.1
- [mysql] Forbid using problematic startup options earliest-offset/specific-offset/timestamp
- [mysql] Validate MySQL server configuration (version, binlog_format, binlog_row_image) before start job
- [mysql] Refactor class names to use 'MySql' prefix instead of 'MySQL'
- [common] Avoid uncompleted state of schema history when engine stop before checkpoint
- [common] Fix schema history run out of memory
- [common] Fix ClassCastException in type conversion
- [docs] Use sphinx-rtd-theme to manage project documents
- [docs] Support versioned documents with RTD(ReadTheDocs)
Upgrade Notes
Please note the following incompatible changes when you're upgrading your existed Flink CDC project.
- The group.id and package name of Flink CDC has renamed from 'com.alibaba.ververica' to 'com.ververica' for making the project
more neutral - The problematic startup options earliest-offset/specific-offset/timestamp has been disabled until they're correctly supported
- The class name prefix has changed from 'MySQL' to 'MySql' for better code style.
Download
Release 1.4.0
Release Notes
Documentation: https://ververica.github.io/flink-cdc-connectors/release-1.4/
Features and Improvements
- [debezium] Bump Flink version to 1.13.0
Download
Release 1.3.0
release-1.2.0
Release Notes
- [common] DebeziumSourceFunction only stores monitored table DDLs which cause error in some situation
- [common] offset state missing if no binlog fetched tile checkpoint after restore
- [common] Bump up Flink version to 1.12.1
- [mysql] Support read from specific MySQL Binlog position
- [mysql] Support "initial", "earliest-offset", "latest-offset" startup mode options
- [mysql] Support read binlog from specific timestamp
- [common] Remove logging security properties
- [common] Bump Debezium version to 1.4.1.Final
- [postgres] Support 'slot.name' option for postgres-cdc connector
- [postgres] Flush LSN offset to PG after checkpoint complete to avoid infinite WAL segments