[Enhancement] Unified Catalog support Paimon #47398

mxdzs0612 · 2024-06-24T09:04:41Z

Why I'm doing:

Make unified catalog support paimon data sources.

example:

CREATE EXTERNAL CATALOG unified
PROPERTIES
(
    "type" = "unified",
    "unified.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://localhost:9083",
    "paimon.catalog.warehouse" = "hdfs://127.0.0.1:9000/paimon",
);

What I'm doing:

Fixes #issue

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

starrocks-cr · 2024-06-24T09:05:27Z

fe/fe-core/src/main/java/com/starrocks/catalog/PaimonTable.java

+        }
+        return properties;
+    }
+
    @Override
    public List<String> getPartitionColumnNames() {
        return partColumnNames;


The most risky bug in this code is:
The getProperties() method may return a mutable reference to the internal properties map, potentially leading to unintended modifications from outside the class.

You can modify the code like this:

@Override public Map<String, String> getProperties() { if (properties == null) { this.properties = new HashMap<>(); } return new HashMap<>(properties); // Return a defensive copy of the properties map }

stephen-shelby · 2024-06-28T03:34:22Z

fe/fe-core/src/main/java/com/starrocks/connector/unified/UnifiedConnector.java

-                DELTALAKE, new DeltaLakeConnector(derivedContext),
-                KUDU, new KuduConnector(derivedContext)
-        );
+        boolean shouldCreatePaimonConnector = null != context.getProperties().get(PAIMON_CATALOG_WAREHOUSE);


why you check this config?

Paimon has multiple checks in its SDK. In the real usage of our customers, it may come to many validation problems when getting metadata in unified catalog even if there is no Paimon table in the catalog. So I add a judgement to filter out Paimon if its warehouse configure does not exist.

if this is necessary, we can move this line to a place just before line 77.

and contains would be better.

letian-jiang

left some comments, overall LGTM.

letian-jiang · 2024-06-30T16:30:46Z

fe/fe-core/src/main/java/com/starrocks/connector/unified/UnifiedConnector.java

-                DELTALAKE, new DeltaLakeConnector(derivedContext),
-                KUDU, new KuduConnector(derivedContext)
-        );
+        boolean shouldCreatePaimonConnector = null != context.getProperties().get(PAIMON_CATALOG_WAREHOUSE);


if this is necessary, we can move this line to a place just before line 77.

letian-jiang · 2024-06-30T16:31:43Z

fe/fe-core/src/main/java/com/starrocks/connector/unified/UnifiedConnector.java

-                DELTALAKE, new DeltaLakeConnector(derivedContext),
-                KUDU, new KuduConnector(derivedContext)
-        );
+        boolean shouldCreatePaimonConnector = null != context.getProperties().get(PAIMON_CATALOG_WAREHOUSE);


and contains would be better.

sonarcloud · 2024-07-01T02:15:53Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

stephen-shelby · 2024-07-02T13:01:47Z

fe/fe-core/src/main/java/com/starrocks/connector/unified/UnifiedConnector.java

+                .put(HUDI, new HudiConnector(derivedContext))
+                .put(DELTALAKE, new DeltaLakeConnector(derivedContext))
+                .put(KUDU, new KuduConnector(derivedContext));
+        boolean containsPaimon = null != context.getProperties().get(PAIMON_CATALOG_WAREHOUSE);


could you move this check to CatalogAnalyzer ?

Seems hard cause it's for unified catalog only. Any ideas?

stephen-shelby · 2024-08-02T06:20:10Z

please rebase main

Signed-off-by: Jiao Mingye <[email protected]>

sonarcloud · 2024-08-03T09:30:35Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-08-03T11:21:41Z

[FE Incremental Coverage Report]

✅ pass : 17 / 20 (85.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/connector/unified/UnifiedMetadata.java	1	2	50.00%	[103]
🔵	com/starrocks/catalog/PaimonTable.java	3	5	60.00%	[47, 48]
🔵	com/starrocks/connector/paimon/PaimonConnector.java	2	2	100.00%	[]
🔵	com/starrocks/connector/unified/UnifiedConnector.java	11	11	100.00%	[]

github-actions · 2024-08-03T11:21:49Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-11-08T09:13:35Z

@Mergifyio backport branch-3.3

github-actions · 2024-11-08T09:13:36Z

@Mergifyio backport branch-3.2

mergify · 2024-11-08T09:13:43Z

backport branch-3.3

✅ Backports have been created

#52744 [Enhancement] Unified Catalog support Paimon (backport #47398) has been created for branch branch-3.3 but encountered conflicts

mergify · 2024-11-08T09:13:44Z

backport branch-3.2

✅ Backports have been created

#52745 [Enhancement] Unified Catalog support Paimon (backport #47398) has been created for branch branch-3.2 but encountered conflicts

## Why I'm doing: Make unified catalog support paimon data sources. example: ```sql CREATE EXTERNAL CATALOG unified PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://localhost:9083", "paimon.catalog.warehouse" = "hdfs://127.0.0.1:9000/paimon", ); ``` Signed-off-by: Jiao Mingye <[email protected]> (cherry picked from commit 2e50513) # Conflicts: # fe/fe-core/src/test/java/com/starrocks/connector/unified/UnifiedMetadataTest.java

## Why I'm doing: Make unified catalog support paimon data sources. example: ```sql CREATE EXTERNAL CATALOG unified PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://localhost:9083", "paimon.catalog.warehouse" = "hdfs://127.0.0.1:9000/paimon", ); ``` Signed-off-by: Jiao Mingye <[email protected]> (cherry picked from commit 2e50513) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/catalog/PaimonTable.java # fe/fe-core/src/main/java/com/starrocks/connector/unified/UnifiedConnector.java # fe/fe-core/src/main/java/com/starrocks/connector/unified/UnifiedMetadata.java # fe/fe-core/src/test/java/com/starrocks/connector/unified/UnifiedConnectorTest.java # fe/fe-core/src/test/java/com/starrocks/connector/unified/UnifiedMetadataTest.java

Make unified catalog support paimon data sources. example: ```sql CREATE EXTERNAL CATALOG unified PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://localhost:9083", "paimon.catalog.warehouse" = "hdfs://127.0.0.1:9000/paimon", ); ``` Signed-off-by: Jiao Mingye <[email protected]> (cherry picked from commit 2e50513)

wangsimo0 · 2024-11-11T03:21:23Z

ignore backport check: 3.3.6

mxdzs0612 requested review from a team as code owners June 24, 2024 09:04

github-actions bot added 3.3 3.2 labels Jun 24, 2024

mergify bot assigned mxdzs0612 Jun 24, 2024

starrocks-cr bot reviewed Jun 24, 2024

View reviewed changes

mxdzs0612 changed the title ~~[Enhancement] Unified Catalog support Paimon table~~ [Enhancement] Unified Catalog support Paimon Jun 24, 2024

mxdzs0612 force-pushed the paimon-unified branch 4 times, most recently from 96da9ff to 0348d6b Compare June 27, 2024 08:28

stephen-shelby reviewed Jun 28, 2024

View reviewed changes

letian-jiang reviewed Jun 30, 2024

View reviewed changes

stephen-shelby reviewed Jul 2, 2024

View reviewed changes

mxdzs0612 added 2 commits August 3, 2024 15:19

unified

fe87eb2

Signed-off-by: Jiao Mingye <[email protected]>

fix comments

0ff519f

Signed-off-by: Jiao Mingye <[email protected]>

mxdzs0612 force-pushed the paimon-unified branch from 7154bd0 to 0ff519f Compare August 3, 2024 09:23

stephen-shelby approved these changes Aug 9, 2024

View reviewed changes

gengjun-git approved these changes Nov 8, 2024

View reviewed changes

gengjun-git merged commit 2e50513 into StarRocks:main Nov 8, 2024
50 checks passed

github-actions bot removed the 3.3 label Nov 8, 2024

github-actions bot removed the 3.2 label Nov 8, 2024

mxdzs0612 deleted the paimon-unified branch November 8, 2024 09:14

mergify bot mentioned this pull request Nov 8, 2024

[Enhancement] Unified Catalog support Paimon (backport #47398) #52744

Closed

42 tasks

mergify bot mentioned this pull request Nov 8, 2024

[Enhancement] Unified Catalog support Paimon (backport #47398) #52745

Closed

42 tasks

mxdzs0612 mentioned this pull request Nov 8, 2024

[Enhancement] Unified Catalog support Paimon (backport #47398) #52749

Merged

24 tasks

mxdzs0612 mentioned this pull request Nov 8, 2024

[Enhancement] Unified Catalog support Paimon (backport #47398) #52750

Merged

24 tasks

github-actions bot added the 3.3 label Nov 11, 2024

caneGuy pushed a commit that referenced this pull request Nov 13, 2024

[Enhancement] Unified Catalog support Paimon (backport #47398) (#52750)

0598898

caneGuy pushed a commit that referenced this pull request Nov 13, 2024

[Enhancement] Unified Catalog support Paimon (backport #47398) (#52749)

04bfa60

github-actions bot added 3.2-merged 3.3-merged labels Nov 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Unified Catalog support Paimon #47398

[Enhancement] Unified Catalog support Paimon #47398

mxdzs0612 commented Jun 24, 2024 •

edited by stephen-shelby

Loading

starrocks-cr bot Jun 24, 2024

stephen-shelby Jun 28, 2024

mxdzs0612 Jun 28, 2024

letian-jiang Jun 30, 2024

letian-jiang Jun 30, 2024

letian-jiang left a comment •

edited

Loading

letian-jiang Jun 30, 2024

letian-jiang Jun 30, 2024

sonarcloud bot commented Jul 1, 2024

stephen-shelby Jul 2, 2024 •

edited

Loading

mxdzs0612 Jul 3, 2024

stephen-shelby commented Aug 2, 2024

sonarcloud bot commented Aug 3, 2024

github-actions bot commented Aug 3, 2024

github-actions bot commented Aug 3, 2024

github-actions bot commented Nov 8, 2024

github-actions bot commented Nov 8, 2024

mergify bot commented Nov 8, 2024 •

edited

Loading

mergify bot commented Nov 8, 2024 •

edited

Loading

wangsimo0 commented Nov 11, 2024

[Enhancement] Unified Catalog support Paimon #47398

[Enhancement] Unified Catalog support Paimon #47398

Conversation

mxdzs0612 commented Jun 24, 2024 • edited by stephen-shelby Loading

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

starrocks-cr bot Jun 24, 2024

Choose a reason for hiding this comment

stephen-shelby Jun 28, 2024

Choose a reason for hiding this comment

mxdzs0612 Jun 28, 2024

Choose a reason for hiding this comment

letian-jiang Jun 30, 2024

Choose a reason for hiding this comment

letian-jiang Jun 30, 2024

Choose a reason for hiding this comment

letian-jiang left a comment • edited Loading

Choose a reason for hiding this comment

letian-jiang Jun 30, 2024

Choose a reason for hiding this comment

letian-jiang Jun 30, 2024

Choose a reason for hiding this comment

sonarcloud bot commented Jul 1, 2024

Quality Gate passed

stephen-shelby Jul 2, 2024 • edited Loading

Choose a reason for hiding this comment

mxdzs0612 Jul 3, 2024

Choose a reason for hiding this comment

stephen-shelby commented Aug 2, 2024

sonarcloud bot commented Aug 3, 2024

Quality Gate passed

github-actions bot commented Aug 3, 2024

[FE Incremental Coverage Report]

file detail

github-actions bot commented Aug 3, 2024

[BE Incremental Coverage Report]

github-actions bot commented Nov 8, 2024

github-actions bot commented Nov 8, 2024

mergify bot commented Nov 8, 2024 • edited Loading

✅ Backports have been created

mergify bot commented Nov 8, 2024 • edited Loading

✅ Backports have been created

wangsimo0 commented Nov 11, 2024

mxdzs0612 commented Jun 24, 2024 •

edited by stephen-shelby

Loading

letian-jiang left a comment •

edited

Loading

stephen-shelby Jul 2, 2024 •

edited

Loading

mergify bot commented Nov 8, 2024 •

edited

Loading

mergify bot commented Nov 8, 2024 •

edited

Loading