[Improment] The issue of indexing large volumes of data tables.

[wg1026688210]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Now we have seen significant improvements in query performance through index creation. However, we have encountered several issues during the indexing process.

1. Some tasks before sorting and writing to Paimon are getting stuck waiting for data for a long time.
2. Due to data skewness, some indexing tasks take a long time to execute.
3. The flink batch Job of building index will global failover after taskManager oom
4. FilesTable can not query the paritions which has built index

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[wg1026688210]

issue 1,2 has been fixed
related
#3081
#2749

issue 3 need a remote suffle service for task failover when using flink