[Feature] Unified catalog support kudu #45590

predator4ann · 2024-05-14T07:52:47Z

Make unified catalog support kudu data sources.

example:

CREATE EXTERNAL CATALOG unified
PROPERTIES
(
    "type" = "unified",
    "unified.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://localhost:9083",
    "kudu.master" = "localhost:7051",
    "kudu.schema-emulation.enabled" = "true",
    "kudu.schema-emulation.prefix" = "impala::"
);

Fixes #45591

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

Signed-off-by: predator4ann <yunlong.sun@hotmail.com>

starrocks-cr · 2024-05-14T07:53:28Z

fe/fe-core/src/main/java/com/starrocks/catalog/KuduTable.java

+        }
+        return properties;
+    }
+
    @Override
    public List<String> getPartitionColumnNames() {
        return partColNames;


The most risky bug in this code is:
The KuduTable constructor does not initialize the properties, masterAddresses, catalogName, databaseName, tableName, and partColNames fields which could lead to NullPointerException when accessing these fields.

You can modify the code like this:

public KuduTable() { super(TableType.KUDU); this.properties = new HashMap<>(); this.masterAddresses = ""; this.catalogName = ""; this.databaseName = ""; this.tableName = ""; this.partColNames = new ArrayList<>(); }

starrocks-cr · 2024-05-14T07:53:38Z

fe/fe-core/src/main/java/com/starrocks/connector/kudu/KuduConnector.java

            HiveMetaClient metaClient = HiveMetaClient.createHiveMetaClient(this.hdfsEnvironment, properties);
-            hiveMetastore = Optional.of(new HiveMetastore(metaClient, catalogName, MetastoreType.HMS));
+            hiveMetastore = Optional.of(new HiveMetastore(metaClient, catalogName, metastoreType));
            // TODO caching hiveMetastore support
        }
        return new KuduMetadata(catalogName, hdfsEnvironment, kuduMaster, schemaEmulationEnabled, schemaEmulationPrefix,


The most risky bug in this code is:
A security issue due to changing visibility of configuration keys from private to public.

You can modify the code like this:

@@ -38,12 +38,14 @@ public class KuduConnector implements Connector { private static final String HIVE = "hive"; + private static final String GLUE = "glue"; private static final String KUDU = "kudu"; - private static final Set<String> SUPPORTED_METASTORE_TYPE = Sets.newHashSet(HIVE, KUDU); - private static final String KUDU_MASTER = "kudu.master"; - private static final String KUDU_CATALOG_TYPE = "kudu.catalog.type"; - private static final String KUDU_SCHEMA_EMULATION_ENABLED = "kudu.schema-emulation.enabled"; - private static final String KUDU_SCHEMA_EMULATION_PREFIX = "kudu.schema-emulation.prefix"; + private static final Set<String> SUPPORTED_METASTORE_TYPE = Sets.newHashSet(HIVE, GLUE, KUDU); + private static final String KUDU_MASTER = "kudu.master"; + private static final String KUDU_CATALOG_TYPE = "kudu.catalog.type"; + private static final String KUDU_SCHEMA_EMULATION_ENABLED = "kudu.schema-emulation.enabled"; + private static final String KUDU_SCHEMA_EMULATION_PREFIX = "kudu.schema-emulation.prefix"; + private static final String DEFAULT_KUDU_MASTER = "localhost:7051"; private final String catalogName; private final String kuduMaster; private final String catalogType;

sonarcloud · 2024-05-14T08:32:33Z

Quality Gate passed

Issues
101 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-05-14T09:53:47Z

[FE Incremental Coverage Report]

✅ pass : 13 / 13 (100.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/connector/unified/UnifiedConnector.java	1	1	100.00%	[]
🔵	com/starrocks/connector/kudu/KuduConnector.java	5	5	100.00%	[]
🔵	com/starrocks/catalog/KuduTable.java	5	5	100.00%	[]
🔵	com/starrocks/connector/unified/UnifiedMetadata.java	2	2	100.00%	[]

github-actions · 2024-05-14T09:54:03Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

miomiocat · 2024-05-20T12:20:58Z

@mergify backport branch-3.2

mergify · 2024-05-20T12:21:06Z

backport branch-3.2

✅ Backports have been created

#45944 [Feature] Unified catalog support kudu (backport #45590) has been created for branch branch-3.2 but encountered conflicts

Signed-off-by: predator4ann <yunlong.sun@hotmail.com> (cherry picked from commit f9cd67d) # Conflicts: # docs/en/data_source/catalog/kudu_catalog.md # docs/zh/data_source/catalog/kudu_catalog.md # fe/fe-core/src/main/java/com/starrocks/catalog/KuduTable.java # fe/fe-core/src/main/java/com/starrocks/connector/kudu/KuduConnector.java # fe/fe-core/src/main/java/com/starrocks/connector/unified/UnifiedConnector.java

miomiocat · 2024-05-20T12:48:34Z

@mergify backport branch-3.3

mergify · 2024-05-20T12:48:39Z

backport branch-3.3

✅ Backports have been created

#45948 [Feature] Unified catalog support kudu (backport #45590) has been created for branch branch-3.3

Signed-off-by: predator4ann <yunlong.sun@hotmail.com> (cherry picked from commit f9cd67d)

Co-authored-by: predator4ann <yunlong.sun@hotmail.com>

eshishki · 2024-05-29T06:57:56Z

i'm using unified catalog with glue, and there is no kudu support there and i'm getting an error now trying to access unified catalog

Null hive.metastore.uris, please check your property's key and value of catalog or resource.
at com.starrocks.common.util.Util.validateMetastoreUris(Util.java:425) ~[starrocks-fe.jar:?]
at com.starrocks.connector.kudu.KuduConnector.getMetadata(KuduConnector.java:98) ~[starrocks-fe.jar:?]
at com.starrocks.connector.unified.UnifiedConnector.lambda$getMetadata$0(UnifiedConnector.java:76) ~[starrocks-fe.jar:?]
at com.google.common.collect.RegularImmutableMap.forEach(RegularImmutableMap.java:297) ~[spark-dpp-1.0.0.jar:?]
at com.starrocks.connector.unified.UnifiedConnector.getMetadata(UnifiedConnector.java:76) ~[starrocks-fe.jar:?]
at com.starrocks.connector.CatalogConnector.getMetadata(CatalogConnector.java:43) ~[starrocks-fe.jar:?]

predator4ann · 2024-05-30T03:12:53Z

i'm using unified catalog with glue, and there is no kudu support there and i'm getting an error now trying to access unified catalog

Null hive.metastore.uris, please check your property's key and value of catalog or resource. at com.starrocks.common.util.Util.validateMetastoreUris(Util.java:425) ~[starrocks-fe.jar:?] at com.starrocks.connector.kudu.KuduConnector.getMetadata(KuduConnector.java:98) ~[starrocks-fe.jar:?] at com.starrocks.connector.unified.UnifiedConnector.lambda$getMetadata$0(UnifiedConnector.java:76) ~[starrocks-fe.jar:?] at com.google.common.collect.RegularImmutableMap.forEach(RegularImmutableMap.java:297) ~[spark-dpp-1.0.0.jar:?] at com.starrocks.connector.unified.UnifiedConnector.getMetadata(UnifiedConnector.java:76) ~[starrocks-fe.jar:?] at com.starrocks.connector.CatalogConnector.getMetadata(CatalogConnector.java:43) ~[starrocks-fe.jar:?]

Apologies for any confusion, but there's no need to check hive.metastore.uris when kudu.catalog.type is configured to glue. The pull request #46436 has been submitted to address this.

[Feature] unified catalog support kudu

d2cea29

Signed-off-by: predator4ann <yunlong.sun@hotmail.com>

predator4ann requested review from a team as code owners May 14, 2024 07:52

github-actions bot added the documentation Improvements or additions to documentation label May 14, 2024

mergify bot assigned predator4ann May 14, 2024

starrocks-cr bot reviewed May 14, 2024

View reviewed changes

predator4ann mentioned this pull request May 14, 2024

[Feature] Unified catalog support Kudu #45591

Open

2 tasks

miomiocat approved these changes May 15, 2024

View reviewed changes

DorianZheng approved these changes May 15, 2024

View reviewed changes

DorianZheng enabled auto-merge (squash) May 15, 2024 07:09

HangyuanLiu approved these changes May 20, 2024

View reviewed changes

DorianZheng merged commit f9cd67d into StarRocks:main May 20, 2024
68 checks passed

github-actions bot added the version:3.4 label May 20, 2024

mergify bot mentioned this pull request May 20, 2024

[Feature] Unified catalog support kudu (backport #45590) #45944

Closed

42 tasks

mergify bot pushed a commit that referenced this pull request May 20, 2024

[Feature] Unified catalog support kudu (#45590)

64ab36b

Signed-off-by: predator4ann <yunlong.sun@hotmail.com> (cherry picked from commit f9cd67d)

mergify bot mentioned this pull request May 20, 2024

[Feature] Unified catalog support kudu (backport #45590) #45948

Merged

42 tasks

miomiocat pushed a commit that referenced this pull request May 20, 2024

[Feature] Unified catalog support kudu (backport #45590) (#45948)

01aa692

Co-authored-by: predator4ann <yunlong.sun@hotmail.com>

github-actions bot added the 3.3-merged label May 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Unified catalog support kudu #45590

[Feature] Unified catalog support kudu #45590

predator4ann commented May 14, 2024 •

edited by miomiocat

starrocks-cr bot May 14, 2024

starrocks-cr bot May 14, 2024

sonarcloud bot commented May 14, 2024

github-actions bot commented May 14, 2024

github-actions bot commented May 14, 2024

miomiocat commented May 20, 2024

mergify bot commented May 20, 2024 •

edited

miomiocat commented May 20, 2024

mergify bot commented May 20, 2024 •

edited

eshishki commented May 29, 2024

predator4ann commented May 30, 2024 •

edited

[Feature] Unified catalog support kudu #45590

[Feature] Unified catalog support kudu #45590

Conversation

predator4ann commented May 14, 2024 • edited by miomiocat

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

starrocks-cr bot May 14, 2024

Choose a reason for hiding this comment

starrocks-cr bot May 14, 2024

Choose a reason for hiding this comment

sonarcloud bot commented May 14, 2024

Quality Gate passed

github-actions bot commented May 14, 2024

[FE Incremental Coverage Report]

file detail

github-actions bot commented May 14, 2024

[BE Incremental Coverage Report]

miomiocat commented May 20, 2024

mergify bot commented May 20, 2024 • edited

✅ Backports have been created

miomiocat commented May 20, 2024

mergify bot commented May 20, 2024 • edited

✅ Backports have been created

eshishki commented May 29, 2024

predator4ann commented May 30, 2024 • edited

predator4ann commented May 14, 2024 •

edited by miomiocat

mergify bot commented May 20, 2024 •

edited

mergify bot commented May 20, 2024 •

edited

predator4ann commented May 30, 2024 •

edited