Skip to content

Release 3.0.0
Compare
Choose a tag to compare
@leonardBang leonardBang released this 07 Dec 03:02
· 377 commits to master since this release
release-3.0.0
ba0d8ba

Release Notes

Documentation: https://ververica.github.io/flink-cdc-connectors/release-3.0

Downloads

Flink CDC

Flink CDC 3.0.0 (tar, asc, sha512)

The binary release can be verified by public key with fingerprint C460A307EAF2701FC9D774EF96719B746D3D0C8A.

Flink CDC Pipeline Connectors

All connectors are release in JAR and available in Maven central repository.

  • Apache Doris pipeline connector 3.0.0 (jar, asc, sha1)
  • MySQL pipeline connector 3.0.0 (jar, asc, sha1)
  • Starrocks pipeline connector 3.0.0 (jar, asc, sha1)
  • Values pipeline connector 3.0.0 (for testing purpose, jar, asc, sha1)

Flink CDC Source Connectors

All connectors are release in JAR and available in Maven central repository.

Features and Improvements

  1. [pipeline-connector][starrocks] Fix char/varchar length inconsistency between cdc and starrocks (#2830)
  2. [cdc-cli][cdc-composer] Use "source-table" and "sink-table" as keywords of route (#2825)
  3. [build][hotfix] Remove duplicate dependency (#2826)
  4. [cdc-cli][cdc-dist] Support loading global config from FLINK_CDC_HOME (#2822)
  5. [cdc-common][hotfix] Remove redundant prefix of pipeline options (#2821)
  6. [cdc-source-connector][oceanbase][e2e] Add OceanBase e2e test case (#2521)
  7. [build] Use flink-shaded-force-shading to force all submodules to generate dependency-reduced-pom.xml
  8. [build] Use ${project.version} instead of ${revision} for dependency version
  9. [build] Skip maven-deploy-plugin for test modules and flink-cdc-dist
  10. [cdc-pipeline-connector][mysql] Fix precision problem of BIT type conversion (#2820)
  11. [cdc-source-connector][mysql] fix deserialization issue on table metadata binlog event (#2682)
  12. [pipeline-connector][mysql] Remove unnecessary serverTimeZone in DebeziumEventDeserializationSchema (#2816)
  13. [build][hotfix] Remove useless suppressions. (#2817)
  14. [cdc-source-connector][mysql] Fix attempted load of isSuspended in MySqlSplitSerializer::deserializeSplit when cdc version <= 2.2.0 (#2550)
  15. [cdc-pipeline-connector][mysql] Ensure the inference of MEDIUMINT_UNSIGNED type matches INT type in table schema (#2811)
  16. [cdc-connector][oceanbase] Add option for obcdc extra configs (#2543)
  17. [pipeline-connector][mysql] Enable send schema change by default (#2815)
  18. [3.0][cdc-runtime] Remove waitForFlushSuccess field in request handler (#2812)
  19. [pipeline-connector][starrocks] Add starrocks pipeline connector (#2765)
  20. [pipeline-connector][doris] add doris pipeline connector. (#2810)
  21. [cdc-pipeline-connector][mysql] Ensure the precision inference of DECIMAL type matches table schema
  22. [3.0][cdc-common & runtime] Remove useless flink imports
  23. [cdc-pipeline-connector][mysql] Add tests for mysql pipeline connector
  24. [cdc-pipeline-connector][mysql] Support more mysql types
  25. [cdc-pipeleine-connecotr][mysql] Send CreateTableEvent in the mysql pipeline connector
  26. [cdc-pipeline-connector][mysql] Parse the alter statement to generate SchemaChangeEvent
  27. [cdc-pipeline-connector][mysql] Introduce mysql cdc pipeline DataSource
  28. [cdc-cli][cdc-composer][cdc-dist] Support submitting job to general Flink environments (#2807)
  29. [cdc-runtime][hotfix] Setup waitFlushSuccess before responding operator to flush. (#2805)
  30. [cdc-common] Introduce "pipeline.local-time-zone" config option which help handle time zone well (#2797)
  31. [cdc-common][hotfix] Remove useless flink imports (#2803)
  32. [build] Let surefire plugin to keep the full stacktrace on test failure (#2804)
  33. [pipeline-common] Use column name to check whether the schema contains specific column or not. (#2801)
  34. [cdc-composer][route][sink] Add name for route and improve name for sink
  35. [cdc-composer][hotfix] Always chain operator by default
  36. [cdc-composer][sink] Use correct name for sink operator
  37. [cdc-runtime] SchemaRegistry should complete the future after making the checkpoint (#2800)
  38. [cdc-connector][sqlserver][tests] Fix UT errors by increasing the wait time after committing SQL (#2799)
  39. [cdc-composer] Add common and runtime JAR when submit (#2795)
  40. [cdc-connector][debezium] Supports conversion from binary bytes to base64 encoded string (#2435)
  41. [cdc-connector][mysql] MysqlTableSource return metadata in a fixed order (#2578)
  42. [cdc-connector][mysql] Fix snapshot fetch size conf does not take effect (#2766)
  43. [cdc-connector][oracle] Expose ability to skip backfill in snapshot period for Oracle CDC (#2793)
  44. [cdc-connector][oracle] Fix wrong table path during backfill phase which leads to potential data loss (#2707)
  45. [cdc-composer][tests] Add integration tests for FlinkPipelineComposer (#2776)
  46. [cdc-dist] Introduce flink-cdc.sh as the entrypoint of the Flink CDC (#2791)
  47. [3.0] [flink-cdc-dist] Collect all distro resources into a TAR (#2636)
  48. [cdc-connector][sqlserver] Expose ability to skip backfill in snapshot period for SqlServer CDC.(#2783)
  49. [cdc-connector][mysql] Expose ability to skip backfill in snapshot period for MySQL CDC.
  50. [cdc-connector][mongodb] Expose ability to skip backfill in snapshot period for Mongodb CDC.
  51. [cdc-connector][postgres] Expose ability to skip backfill in snapshot period for Postgres CDC.
  52. [cdc-connector][base] Support skip snapshot backfill for incremental snapshot connector
  53. [cdc-connector][base][tests] Introduce snapshot phase hooks for better snapshot test
  54. [cdc-composer] Add routing logic into the Flink composer
  55. [cdc-cli][cdc-composer] Use sourceTable and sinkTable as keyword in route definition
  56. [cdc-runtime] Add RouteFunction for routing change events
  57. [cdc-common] Introduce AssertJ-style assertion system for Flink CDC data structures
  58. [hotfix][mysql][tests] Avoid canceling a finished job in tests of SpecificStartingOffsetITCase (#2687)
  59. [docs][zh][hotfix] Remove redundant sections from mysql-cdc documentation (#2777)
  60. [mongodb][hotfix] Fix MongoDB's rewriteOutputBuffer to emit +I rather than +U (#2760) (#2760)
  61. [cdc-common] Introduce createFieldGetters method in SchemaUtils to build FieldGetters of given Schema. (#2762)
  62. [cdc-composer] use DataSinkWriterOperatorFactory to replace all SinkWriterOperatorFactory. (#2773)
  63. [cdc-pipeline-connector][values] Avoid using default as name of namespace and schema. (#2774)
  64. [cdc-runtime] Improve DataSinkWriterOperator logic that only emit latest schema for events that are not CreateTableEvent (#2771)
  65. [cdc-common] add PublicEvolving annotation to SchemaUtils (#2756)
  66. [3.0][cdc-composer] Introduce pipeline.name config to support custom flink job name (#2768)
  67. [cdc-base] Extract the common parts of each ScanFetchTask into AbstractScanFetcherTask.
  68. [docs][hotfix] Configuration 'slot.name' is required in mysql-postgres-tutorial-zh.md (#2763)
  69. [cdc-runtime] Improve partitioning logics and add tests for PrePartitionOperator(#2761)
  70. [cdc-runtime][tests] Introduce Harness test utilities for CDC customized operators
  71. [cdc-runtime] Make sure CreateTableEvent is always sent before DataChangeEvent even if during restoration (#2767)
  72. [docs][hotfix] Configuration 'slot.name' is required in mysql-postgres-tutorial.md (#2764)
  73. [oceanbase] Use global timestamp to start log client (#2565)
  74. [cdc-pipeline-connector][values] Add event set MULTI_SPLITS_SINGLE_TABLE to mock source with multiple splits (#2737)
  75. [hotfix][cdc-composer] Use correct factory to discovery DataSink (#2757)
  76. [3.0][cdc-common] Introduce BinaryRecordDataGenerator to help to create BinaryRecordData
  77. [3.0][cdc-common] Use BinaryRecordData in DataChangeEvent instead of GenericRecordData
  78. [3.0][cdc-common] Introduce binaryRecordData to avoid type serialization and improve performance as well
  79. [build][hotfix] Use the correct format for labeler.yml (#2758)
  80. [build] Add github labeler for pull request
  81. [docs] Add DataStream application package guidance (#2583)
  82. [cdc-common] Introduce TableFilter to filter table objects from given TableId pattern (#2673)
  83. [cdc-composer] Initialize context with existing config of SourceDef and SinkDef (#2749)
  84. [hotfix][docs] Adjust the correct doc versions (#2679)
  85. [docs][oracle] Fix oracle cdc quick start (#2727)
  86. [cdc-runtime] Use CollectCoordinationResponse as a wrapper of CoordinationResponse
  87. [cdc-common] MetadataApplier should extends Serializable
  88. [cdc-runtime] Rewrite DataSinkWriterOperator to avoid classloading issues using reflection
  89. [cdc-composer] Implementation of Flink pipeline composer for chaining everything together
  90. [cdc-composer] Introduce partitioning related runtime functions and translator
  91. [cdc-composer] Introduce data sink and schema operator translators
  92. [cdc-composer] Introduce DataSourceTranslator for composing source
  93. [cdc-runtime] Introduce schema manager for managing schema version and handling schema changes
  94. [cdc-composer] Add FlinkEnvironmentUtils for adding JAR to StreamExecutionEnvironment
  95. [cdc-common] Define pipeline options and schema change behaviors
  96. [cdc-common] Support adding Column directly in schema builder
  97. [hotfix][vitess] Fix invalidate package name
  98. [cdc-pipeline-connector][values] Introduce values cdc pipeline connector
  99. [3.0][cdc-common] Introduce TableSchemaState and its Serializer for DataSink to hold the table schemas.
  100. [3.0][cdc-common][minor] Add toString method for some classes.
  101. [3.0][cdc-common] Add SchemaUtil to perform schema evolution (#2732)
  102. [3.0][cdc-runtime] Provide SchemaOperator and SchemaRegistry to handle schema changes (#2685)
  103. [sqlserver] Fix timestamp type parse of sqlserver (#2708)
  104. [3.0][cdc-runtime] Add TypeInformation for Event class (#2728)
  105. [3.0][cdc-runtime][tests] Add tests for type serializers
  106. [3.0][cdc-runtime][hotfix] Move type serializers to runtime module
  107. [3.0][cdc-common] Introduce data type for record field in DataChangeEvent
  108. [3.0][cdc-common][minor] Improve public methods for schema event
  109. [3.0][cdc-common][minor] Move FlushEvent to common module
  110. [mongodb] Incremental snapshot source supports comma-separated format parameter for databases and collections (#2724)
  111. [3.0][cdc-common] Use built-in ConfigOption for cdc-common module (#2716)
  112. [3.0][cdc-common] Add ConText for Source/Sink initialization. (#2672)
  113. [3.0][cdc-common] Add TypeSerializer implementations (#2683)
  114. [build] Add header copyright check for java files
  115. [common] Introduce Configuration and related utilities (#2681)
  116. [oracle] Skip unsupported nested table for legacy debezium source function
  117. [oracle] Skip unsupported nested table for Incremental OracleSource
  118. [3.0][common] Use built-in API annotations to make common module free with flink
  119. [3.0][common] Introduce built-in API annotations for project
  120. [build] Bump Flink version to 1.18.0 (#2463)
  121. [build][tests] Enable CI tests for CDC 3.0 modules (#2678)
  122. [3.0][cdc-common] Introduce internal data structures and generic implementations (#2688)
  123. [composer] Connector factory and JAR discovery utilities (#2662)
  124. [postgres-cdc] Add tests for latest-offset startup strategy (#2527)
  125. [3.0][cdc-common] Change DataChangeEvent from interface to class, and add implementation for DataChangeEvent. (#2677)
  126. [build][tests] Enable CI tests for CDC 3.0 modules
  127. [3.0][cdc-runtime] add DataSinkWriterOperator to process Event (#2649)
  128. [3.0][cdc-common] Add SchemaChangeEvent implementations (#2664)
  129. [cli] Implementation of CLI frontend (#2660)
  130. [docs] Remove flink 1.13 from supported flink versions
  131. [3.0][cdc-common] Add GenericRecordData implementation for RecordData
  132. [3.0][cdc-common] Keep columns in Schema in fixed order
  133. [3.0][cdc-common] Use RecordData to represent before and after record.
  134. [build] Introduce -Dfast property for skipping code style checks
  135. [test] Bump maven-surefire-plugin version to 3.0.0-M5
  136. [test] Introduce JUnit 5 and AssertJ dependencies
  137. [flink-cdc-composer] Base interfaces and models for composer and pipeline definition (#2656)
  138. [3.0][cdc-common] Introduce Connector Factory APIs (#2661)
  139. [3.0][cdc-common] Add getTableSchema method for MetadataAccessor (#2650)
  140. [3.0][cdc-common] Introduce Schema API for CDC 3.0
  141. [3.0][cdc-common] Introduce Event DataSink APIs for Flink CDC 3.0
  142. [3.0][cdc-common] Introduce DataSource related APIs for Flink CDC 3.0
  143. [pom] Use ${revision} replace specific version. (#2623)
  144. [3.0][cdc-runtime] introduce schema evolution framework API
  145. [3.0][cdc-common] Introduce data types for cdc 3.0
  146. [3.0][cdc-common] Introduce Event related APIs for Flink CDC 3.0
  147. [3.0][kickoff] Introduce basic modules for Flink CDC 3.0
  148. [mongodb] Use SampleBucketSplitStrategy for shard collection with hashed keys (#2595)
  149. [vitess][hotfix] Use stable image tag for vitess docker image (#2594)
  150. [oracle] Bump oracle docker image version to 19c
  151. [oracle] Refactor oracle unit test
  152. [build] Bump testcontainers version and polish dependency
  153. [mysql] Use automatic resource management to replace 'try - finally' code block. (#2552)
  154. [sqlserver] Fix get latest lsn (#2551)
  155. [flink-connector-debezium] Support overriding some debezium properties in DebeziumSourceFunction (#2530)
  156. [e2e] Remove db docker image after test cases are executed. (#2508)
  157. [postgres] Reuse pool connection based on database in postgres cdc
  158. [mysql] remove redundant source metrics processTime and emitDelay in mysql source
  159. [cdc-base]remove redundant source metrics register(#2403)
  160. [cdc-base] JdbcConnectionPools supports multiple JDBC sources
  161. [tests][e2e] add log config to e2e tests (#2464)
  162. [postgres] Not drop replication slot for stream split (#2436)
  163. [postgres] Close jdbc connection after creating replication slot for stream split (#2425)
  164. [test] Close resources after each test (#2438)
  165. [hotfix][docs] Fix the abnormal display of pictures (#2513)
  166. [cdc-base] Fix TM hangs caused by uncaught exception (#2511)
  167. [postgres] Close idle readers when snapshot finished (#2400)
  168. [oceanbase][ci] Move the OceanBase CI to free azure pipeline (#2506)
  169. [mongodb][hotfix] Fix Mongo collection regex match logic (#2503)
  170. [hotfix][mysql] Keep assigned splits in order to fix wrong meta group calculation
  171. [mysql] Add finished unack splits to state for the MysqlSourceReader (#2399)
  172. [mongodb][hotfix] Mongo CDC fails to capture collections with . in names (#2488)
  173. [e2e][tidb] Split e2e tests into two groups to decrease the disk usage for per azure job to avoid tidb disk full error
  174. [hotfix][e2e][vitess] Fix module name typo
  175. [e2e] Remove flink 1.13 tests from e2e test matrix
  176. [cdc-base] Fix parsing error when serializing and deserializing the table name with dot (#2443)
  177. [oracle] Correct the naming error (#2405)
  178. [docs] Remove unreachable images in README (#2426)
  179. [docs] Add contribution guidance section in document (#2411)
  180. [sqlserver] Add support bigint type as table split column (#2416)
  181. [hotfix][sqlserver] Fix backfill stream task hang (#2374)
  182. [hotfix][mysql] Trim default value for BIGINT and SMALLINT types when parsing MySQL DDL (#2418)
  183. [sqlserver] Add table filter to speed up SqlServerSchema read (#2369)
  184. [Doc] Use redo log instead of binlog in Oracle document (#2408)
  185. [mysql] Filter databases that do not need to be read when discovering tables (#2160)
  186. [test] Fix TiDBE2eITCase by updating the container version to avoid tikv disk full failure (#2402)
  187. [test][postgres] Close postgres containers after tests and fix container reuse bug (#2394)
  188. [mysql] quota the field names when executing mysql query (#2381) (#2388)
  189. [test][mysql] Provide the server-time-zone setting in MySqlSourceExampleTest#testConsumingAllEvents (#2364)
  190. [hotfix][mysql] Catch underlying throwable in snapshot reading.
  191. [hotfix][mysql] Fix failed test LegacyMySqlSourceTest#testSnapshotOnFailedSource (#2384)
  192. [db2] [tests] Fix unstable Db2ConnectorITCase due to cleanup failure
  193. [vitess] Improve build changelog mode (#2355)
  194. [mongodb][tests] Improve test coverage & add class visibility annotations
  195. [mongodb] Add pendingRecords and numRecordsInErrors metrics for source
  196. [mongodb][hotfix] Fix SourceRecord range check (#2343)
  197. [polish] Add type parameter for MySqlRecords(#2300)
  198. [doc] fix errors in PostgresParallelSourceExample (#2324)
  199. [docs][oracle] Add an example of Incremental Snapshot based DataStream for Oracle CDC Connector
  200. [mongodb] Add disableCursorTimeout option (#2332)
  201. [mongodb] [tests] Loosen E2E timeout limitations
  202. [vitess] Migrate connector properties to debezium 1.9.7 (#2260)
  203. [hotfix][sqlserver] Fix sqlserver monitor same table in other database (#2336)
  204. [debezium] Fix DebeizumSourceFunction can not do savepoint after close (#2259)
  205. [Mysql] Fix assigning duplicate snapshot splits when enable scan newly added tables (#2096)
  206. [mysql] skip closing reader when the reader received the binlog split (#2261)
  207. [MySQL] Optimize how to construct config table.include.list (#2274)
  208. [oracle] Fix class conflict when using multi cdc sql connector
  209. [hotfix][doc] Fix doc site with 404 (#2322)
  210. [docs] Add connector features table to help users quickly understand the supported features of each connector
  211. [mysql] Remove the finished snapshot splits for the binlog split when restoring from the checkpoint (#2292)
  212. [hotfix][sqlserver] Fix sqlserver close idle and chunk key column missing (#2305)
  213. [mongodb] [docs] Adds scanFullChangelog docs (#2294)
  214. [mongodb] Supports scanFullChangelog feature & deprecate copyExisting option
  215. [mongodb] Support MongoDB 6.0+ & update testcases
  216. [postgres] Fix the slot name conflict bug (#2251) (#2281)
  217. [docs] Update Postgres CDC documents for required slot.name option (#2270)
  218. [build] Add vcs.xml for IDEA (#2064)
  219. [docs] add docs for the mysql tables without primary keys (#2227)
  220. [MySQL] Fix the startup mode exception message (#2264)
  221. [oracle] Use oracle connection in context for each reader subtask (#2254) (#2258)
  222. [hotfix] Update support version (#2255)
  223. [hotfix] update dbz option name 'xx. whitelist' to 'xx.include.list'(xx = database, schema or table) (#2212)

Contributor List (sorted alphabetically)

  1. BIN
  2. Dian Qi
  3. EchoLee5
  4. FlechazoW
  5. FocusComputing
  6. Hang Ruan
  7. He Wang
  8. Hongshun Wang
  9. Jiabao Sun
  10. Josh Mahonin
  11. Kunni
  12. Leonard Xu
  13. Maciej Bryński
  14. Malcolmjian
  15. North.Lin
  16. Paddy Gu
  17. PengFei Li
  18. Qingsheng Ren
  19. Shawn Huang
  20. Simonas Gelazevicius
  21. Sting
  22. Tyrantlucifer
  23. TJX2014
  24. Xin Gong
  25. baxinyu
  26. chenlei677
  27. e-mhui
  28. empcl
  29. gongzhongqiang
  30. gaotingkai
  31. ice
  32. joyCurry30
  33. l568288g
  34. lvyanquan
  35. pgm-rookie
  36. rookiegao
  37. skylines
  38. syyfffy
  39. van
  40. wudi
  41. wuzhenhua
  42. yunqingmo
  43. yuxiqian
  44. zhaomin
  45. zyi728
  46. zhangjian
Assets 5
Loading