[Feature] auto create new tag for flink checkpoint

[MonsterChenzhuo]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Paimon's auto-tag didn't get created yesterday. Could it be that it won't trigger if there are no data changes? Is it triggered on write? As expected, it's triggered on write. I manually inserted a few rows of data, and the tags were generated.

This can't be used in a batch processing scenario because batch processing requires an accurate trigger signal. When no data is written into the stream table, the auto-generation of tags is interrupted. Therefore, we need a mechanism for generating tags based on checkpoints. This mechanism should take into account settings for parameters like 'tag.automatic-creation', 'tag.creation-period', 'tag.creation-delay', and 'tag.num-retained-max'.

• Decouple tag Auto Creation logic [flink] Decouple tag Auto Creation logic #1940
• auto create new tag for flink checkpoint [flink] Auto create new tag for flink checkpoint #1949

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!