[core] support custom commitUser #3474
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wwj6591812
force-pushed
the
support_custom_commitUser_0603
branch
from
e4b11cf
to
615d2a6
Compare
JingsongLi
requested changes
Jun 5, 2024
|
It is not so simple... The commitIdentifier should be incremental too. |
wwj6591812
force-pushed
the
support_custom_commitUser_0603
branch
from
615d2a6
to
4bd1e3c
Compare
Yes，I change the config from customCommitUser to customCommitUserPrefix. |
JingsongLi
reviewed
Jun 6, 2024
JingsongLi
reviewed
Jun 6, 2024
wwj6591812
force-pushed
the
support_custom_commitUser_0603
branch
from
4bd1e3c
to
c253c4e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
In our business scenario, the full data are written to the paimon table through bound stream job. Using of bound stream job instead of batch job is to enable downstream build index faster and further reduce the inc data of catchup. When bound stream job failover, the random commitUser will recover from Flink state, this is no problem.
But In our business scenario, we use Apache Airflow schedule Flink jobs. A Airflow node will fail due to lots of reasons. Then user clear the airflow node, a new Flink bound stream job will start with no state, so a new CommitUser will be generated. This will affect downstream judgment on whether to use a new build index processor.
So I submit this pr, we can assign a fixed commitUser to an Airflow node.
Tests
Add ut
API and Format
Documentation