Skip to content

[spark] Expose external table type to Spark#8306

Merged
JingsongLi merged 5 commits into
apache:masterfrom
huangxiaopingRD:codex/external-table-type-display
Jun 24, 2026
Merged

[spark] Expose external table type to Spark#8306
JingsongLi merged 5 commits into
apache:masterfrom
huangxiaopingRD:codex/external-table-type-display

Conversation

@huangxiaopingRD

Copy link
Copy Markdown
Contributor

Purpose

Fix an inconsistency where Spark did not recognize Paimon external tables as external in Spark-facing metadata.
Before this change, a Paimon table could already be stored as table.type=EXTERNAL, but Spark-side logic relies on the reserved property TableCatalog.PROP_EXTERNAL to classify a table as external. Because that property was not exposed from PaimonSparkTableBase, Spark metadata output could report the wrong table type, such as showing a Paimon external table as managed in DESC FORMATTED.
This change fixes that by exposing PROP_EXTERNAL from PaimonSparkTableBase when the underlying Paimon table is external. That keeps Spark behavior consistent with the existing property-based contract already used by Spark-side commands like SHOW TABLE EXTENDED.

Tests

Adds a focused Hive catalog test to verify that:
DESC FORMATTED shows EXTERNAL for external tables
DESC FORMATTED shows MANAGED for managed tables

@huangxiaopingRD huangxiaopingRD force-pushed the codex/external-table-type-display branch from 5858cbe to 582d63b Compare June 20, 2026 16:24
@JingsongLi

Copy link
Copy Markdown
Contributor

Please fix tests

@huangxiaopingRD

huangxiaopingRD commented Jun 23, 2026

Copy link
Copy Markdown
Contributor Author

Please fix tests

Test cases passed, please review it again. Thanks! @JingsongLi

@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi JingsongLi merged commit 5d3752c into apache:master Jun 24, 2026
12 checks passed
@huangxiaopingRD huangxiaopingRD deleted the codex/external-table-type-display branch June 24, 2026 07:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants