[Data] New Catalog Abstraction and Unity Catalog Read Consolidation

[kyuds]

Description

We have several read_* apis for different sources like Parquet, Delta, and Iceberg. At the same time, users also use catalog providers like Unity Catalog.

For Unity Catalog in particular, we have a separate function called read_unity_catalog which performs catalog authentication and then offloads to the appropriate data format (eg: Parquet). This indirection is a bit confusing, and to pass in arguments for the offloaded function, users have to pass in a dict in reader_kwargs where there is no schema, etc defined.

This PR introduces a new Catalog contract and a UnityCatalog implementation that users can pass into read_* functions and authenticate right away.

This also provides basis to move the authentication part from the Dataset construction phase to the execution phase, which will also potentially allow Ray workers to retry upon credential expiration.

@PublicAPI(stability="alpha")
class Catalog(ABC):
    """A directory service that resolves a table name to a readable source."""

    @abstractmethod
    def resolve(self, table: str, *, reader: ReaderFormat) -> ResolvedSource:
        """Resolve ``table`` for the given ``reader``."""
        ...

TODO:

• Add documentation

Test (will mark complete if pass)

• Reading parquet data with unity catalog works
• Reading delta data with unity catalog works
• Reading iceberg data with unity catalog works
• read_unity_catalog still works and data format inference also works

Related issues

N/A

Additional information

Note:

• The diff is mainly porting over code from removed files.
• Deprecated read_unity_catalog api.

[gemini-code-assist]

Code Review

This pull request introduces a new Catalog connector API (Catalog, UnityCatalog) to invert the dependency between Ray Data readers and the authentication layer, allowing readers like read_parquet, read_delta, and read_iceberg to accept an optional catalog argument for credential and path resolution. The legacy read_unity_catalog function is deprecated and refactored into a shim. The review feedback suggests several important improvements: tracking and cleaning up active temporary GCP credential files to prevent resource leaks and accumulating atexit handlers, and adding defensive type checks in read_parquet and read_delta to ensure that the input path is a single string table identifier when a catalog is specified.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ef239dc. Configure here.}