[cdc] paimon cdc support snapshot mode

[MOBIN-F]

Purpose

Currently paimon cdc does not support scan.startup.mode=snapshot, but flink-cdc 3.x(mysql、pg、mongdb cdc) already supports shapshot mode

Tests

API and Format

Documentation

[JingsongLi]

+1

[LinMingQiang]

Why MysqlSource can't finish when i set -Dexecution.runtime-mode=BATCH .

If in scan.startup.mode=snapshot mode , 'execution.runtime-mode' should/can be Batch?

[LinMingQiang]

flink run -Dexecution.runtime-mode=BATCH \
paimon-flink-action-1.0-SNAPSHOT.jar \
mysql_sync_table \
--warehouse xxx \
--database local_database \
--table test_paimon_cdc_opt \
--mysql_conf hostname=localhost \
--mysql_conf username=root \
--mysql_conf password=123456789 \
--mysql_conf database-name=paimon \
--mysql_conf table-name=paimon_cdc_opt \
--mysql_conf scan.startup.mode=snapshot \
--table_conf bucket=10 \
--table_conf tag.automatic-creation=watermark \
--partition_keys dt \
--primary_keys dt,id \
--type-mapping char-to-string```

[LinMingQiang]

Just 1 record in mysql. and operator ```Source: MySQL Source -> Parse -> Side Output```` will be never finish.

[MOBIN-F]

Why MysqlSource can't finish when i set -Dexecution.runtime-mode=BATCH .

If in scan.startup.mode=snapshot mode , 'execution.runtime-mode' should/can be Batch?

hi @LinMingQiang , no need to set the batch mode, the job will automatically complete and exit

2024-09-10 18:20:40,006 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job MySQL-Paimon Table Sync: rt_ods.xxxx(d9998e2d7bfccd5d320fb4ac21761c3d) switched from state
 RUNNING to FINISHED.
-> 
2024-09-10 18:20:40,007 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - ExecutionGraph d9998e2d7bfccd5d320fb4ac21761c3d reached terminal state FINISHED.
-> 
2024-09-10 18:20:40,343 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Shutting YarnJobClusterEntrypoint down with application status SUCCEEDED. Diagnostics null.

[LinMingQiang]

It is normal in mode streaming. My question is why can't I use batch mode?

[LinMingQiang]

In addition, the job will not finish until checkpoint is triggered once in streaming mode.

[MOBIN-F]

@LinMingQiang See more details in discussion of #2867.

[LinMingQiang]

I have reported the bug to cdc :

https://issues.apache.org/jira/browse/FLINK-36280