Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a parameter that controls the number of StreamLoad tasks committed per partition #92 #99

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

Add a parameter that controls the number of StreamLoad tasks committed per partition #92 #99

wants to merge 10 commits into from

Commits on Apr 6, 2023

  1. 新增分区任务原子性控制参数，参数默认值为false,若为false,则为每个分区可执行多次StreamLoad,若为true,则限制每个分… 

    …区只能执行一次StreamLoad,即将一个分区的所有数据通过一个StreamLoad任务写入Doris。避免task任务失败重试时对同样的数据提交多次StreamLoad任务。
    @baishaoisde
    baishaoisde committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    3a1aed9 View commit details
    Browse the repository at this point in the history

Commits on May 17, 2023

  1. fix

    @baishaoisde
    baishaoisde committed May 17, 2023
    Configuration menu
    Copy the full SHA
    9b0efff View commit details
    Browse the repository at this point in the history

Commits on May 29, 2023

  1. Merge branch 'master' into baishaode_pr

    @baishaoisde
    baishaoisde committed May 29, 2023
    Configuration menu
    Copy the full SHA
    8e87ed2 View commit details
    Browse the repository at this point in the history

Commits on May 30, 2023

  1. [refactor] Unified writing through DorisWriter (apache#104) 

    * use writer to write data
* resolve conflicts
* unify jackson version
* remove useless code
    @gnehil
    gnehil committed May 30, 2023
    Configuration menu
    Copy the full SHA
    4538a43 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2023

  1. Update version (apache#106)

    @gnehil
    gnehil committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    84dbb2c View commit details
    Browse the repository at this point in the history

  2. Update pom.xml (apache#107)

    @caoliang-web
    caoliang-web committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    9291a2e View commit details
    Browse the repository at this point in the history

  3. Pom update (apache#108)

    @JNSimba
    JNSimba committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    53525c2 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2023

  1. Added atomicity control parameters for partition task, Parameter defa… 

    …ults to false. If false, StreamLoad can be performed multiple times per partition. If true, limit StreamLoad to one time per partition, that is, all the data of a partition is written to Doris via a StreamLoad task. Avoid submitting multiple StreamLoad tasks with the same data when the task fails to retry.
    @baishaoisde
    baishaoisde committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    83fdd53 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/baishaode_pr' into baishaode_pr 

    # Conflicts:
#	spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisSourceProvider.scala
#	spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisStreamLoadSink.scala
    @baishaoisde
    baishaoisde committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    734ef75 View commit details
    Browse the repository at this point in the history

  3. resolve conflict

    @baishaoisde
    baishaoisde committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    6e1f6bb View commit details
    Browse the repository at this point in the history