Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] [connector-cdc-oracle ] Unsupport read partition table #8229

Open
2 of 3 tasks
Ashal opened this issue Dec 6, 2024 · 2 comments · May be fixed by #8265
Open
2 of 3 tasks

[Bug] [connector-cdc-oracle ] Unsupport read partition table #8229

Ashal opened this issue Dec 6, 2024 · 2 comments · May be fixed by #8265
Labels
bug help wanted

Comments

@Ashal
Copy link

Ashal commented Dec 6, 2024

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

oracle实时监听不到分区表,历史数据也不会同步，实时数据也不会监听
image
在flinkcdc中已经修复该问题
image

SeaTunnel Version

2.3.7，2.3.8

SeaTunnel Config

env {
  job.mode = "STREAMING"
  parallelism = 1
  checkpoint.interval = 10000
  read_limit.rows_per_second=40960
}
source {
 Oracle-CDC {
    base-url = "jdbc:oracle:thin:@//10.9.3.71:1521/DEV"
    username = "logminer_erp"
    password = "NxJ89TX0Bbc"
    database-names = ["DEV"]
    schema-names = ["XLA"]
    table-names = ["DEV.XLA.XLA_AE_LINES"]
    table-names-config = [
      {
        table = "DEV.XLA.XLA_AE_LINES",
        primaryKeys = ["APPLICATION_ID","AE_HEADER_ID","AE_LINE_NUM"]
      }

    ]

    source.reader.close.timeout = 120000
    skip_analyze = true
    use_select_count = true
    exactly_once = false
  }
}


sink {
  Doris {
    fenodes = "zxxx"
    username = root
    password = "888888g"
    database = "oracle_cdc"
    table = "${table_name}"
    sink.label-prefix = "prod-oracle"
    sink.enable-2pc = "false"
    sink.enable-delete = "true"
    generate_sink_sql = true
    doris.batch.size = 40960
    doris.config {
      format = "json"
      read_json_by_line = "true"
    }
    schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"
    data_save_mode = "APPEND_DATA"
    save_mode_create_template = """
      CREATE TABLE IF NOT EXISTS `${database}`.`${table_name}` (
      ${rowtype_primary_key},
      ${rowtype_fields}
      ) ENGINE=OLAP
        UNIQUE KEY (${rowtype_primary_key})
        DISTRIBUTED BY HASH (APPLICATION_ID)
        PROPERTIES (
          "replication_allocation" = "tag.location.default: 1",
          "in_memory" = "false",
          "storage_format" = "V2",
          "disable_auto_compaction" = "false"
        )
      """
  }
}

Running Command

集群模式启动

Error Exception

无错误

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@Ashal Ashal added the bug label Dec 6, 2024
@hailin0 hailin0 changed the title [Bug] [connector-cdc-oracle ] oracle实时监听不到分区表 [Bug] [connector-cdc-oracle ] Not support read partition table Dec 6, 2024
@hailin0 hailin0 changed the title [Bug] [connector-cdc-oracle ] Not support read partition table [Bug] [connector-cdc-oracle ] Unsupport read partition table Dec 6, 2024
@zhangshenghang zhangshenghang linked a pull request Dec 10, 2024 that will close this issue
Draft
4 tasks
@tyansc
Copy link

tyansc commented Dec 16, 2024

遇到同样的问题，可以正常读取oracle的非分区表，而分区表会直接报错，有没有解决办法？

2024-12-16 14:31:42,542 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Fatal Error,

2024-12-16 14:31:42,542 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues

2024-12-16 14:31:42,542 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Reason:SeaTunnel job executed failed

2024-12-16 14:31:42,543 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:213)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: java.lang.NullPointerException
at org.apache.seatunnel.connectors.cdc.base.source.enumerator.IncrementalSplitAssigner.createIncrementalSplit(IncrementalSplitAssigner.java:240)
at org.apache.seatunnel.connectors.cdc.base.source.enumerator.IncrementalSplitAssigner.createIncrementalSplits(IncrementalSplitAssigner.java:197)
at org.apache.seatunnel.connectors.cdc.base.source.enumerator.IncrementalSplitAssigner.getNext(IncrementalSplitAssigner.java:106)
at org.apache.seatunnel.connectors.cdc.base.source.enumerator.IncrementalSourceEnumerator.assignSplits(IncrementalSourceEnumerator.java:172)
at org.apache.seatunnel.connectors.cdc.base.source.enumerator.IncrementalSourceEnumerator.run(IncrementalSourceEnumerator.java:70)
at org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask.stateProcess(SourceSplitEnumeratorTask.java:323)
at org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask.call(SourceSplitEnumeratorTask.java:141)
at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:693)
at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1018)
at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:39)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

    at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:205)
    ... 2 more

@davidzollo
Copy link
Contributor

If anyone who'd like to help fix this issue, please leave a message here. Thank you!
Here are some helpful resources to get started:

By the way, since this might be your first contribution, I think we can have a more in-depth communication to help you get familiar with SeaTunnel. Feel free to contact me on LinkedIn (David Zollo) or WeChat (taskflow). When adding me, please let me know your GitHub ID. I'd be happy to assist you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug help wanted
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[hotfix] [connector-cdc-oracle ] support read partition table zhangshenghang/data-integration
4 participants
@Ashal @hailin0 @davidzollo @tyansc