Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] yarn-session模式,无法同步数据,但没有报错异常 #324

Open
2 of 3 tasks
lliuql opened this issue Feb 29, 2024 · 1 comment
Open
2 of 3 tasks

Comments

@lliuql
Copy link

lliuql commented Feb 29, 2024

Search before asking

  • I had searched in the issues and found no similar issues.

Version

java version "1.8.0_212"
flink-1.17.1
hadoop3.3.4
下面三个jar都试过了
-rw-rw-r--. 1 liuqh liuqh 8452171 2月 23 17:28 flink-doris-connector-1.17-1.4.0.jar
-rw-rw-r--. 1 liuqh liuqh 11915843 2月 23 17:28 flink-doris-connector-1.17-1.5.0.jar
-rw-rw-r--. 1 liuqh liuqh 12193974 2月 23 17:28 flink-doris-connector-1.17-1.5.1.jar

What's Wrong?

在Standalone与Yarn Application模式下都能正常运行。但在Yarn Session模式下,数据同步存在问题。例如,当MySQL库中的test表有5条数据时,开启整库同步后,虽然成功创建了test表到Doris,但doris表数据为0条。随后对MySQL中的test表结构修改,表结构能够更新到Doris,但doris的test数据依然是0条

What You Expected?

成功同步数据

How to Reproduce?

`

1.5.0yarn-session没办法删除数据,但是可以同步表结构

bin/flink run
-Dexecution.checkpointing.interval=10s
-Dstate.checkpoints.dir=hdfs://hadoop102:8020/flink/checkpoints/report_sync
-Dstate.savepoints.dir=hdfs://hadoop102:8020/flink/flink-savepoints/report_sync
-Dparallelism.default=1
-c org.apache.doris.flink.tools.cdc.CdcTools
job_jar/flink-doris-connector-1.17-1.5.0.jar
mysql-sync-database
--database test
--mysql-conf
server-id=5310-5313
--mysql-conf hostname=192.168.100.128
--mysql-conf port=3306
--mysql-conf username=root
--mysql-conf password='123456'
--mysql-conf database-name=test
--sink-conf fenodes=192.168.88.120:8030
--sink-conf username=root
--sink-conf password='Ryuu1234@'
--sink-conf jdbc-url=jdbc:mysql://192.168.88.120:9030
--sink-conf sink.label-prefix=testx_sync2
--table-conf replication_num=1
--including-tables "test|test_tb01"
`

Anything Else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@JNSimba
Copy link
Member

JNSimba commented Jul 11, 2024

You can check the configuration of jm and tm on flink web to see if the checkpoint and other information have been passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants