[SPARK-15777] Catalog federation

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Reynold Xin
• Created: 06/Jun/16 07:53
• Updated: 08/Oct/19 05:42
• Resolved: 08/Oct/19 05:42

Description

This is a ticket to track progress to support federating multiple external catalogs. This would require establishing an API (similar to the current ExternalCatalog API) for getting information about external catalogs, and ability to convert a table into a data source table.

As part of this, we would also need to be able to support more than a two-level table identifier (database.table). At the very least we would need a three level identifier for tables (catalog.database.table). A possibly direction is to support arbitrary level hierarchical namespaces similar to file systems.

Once we have this implemented, we can convert the current Hive catalog implementation into an external catalog that is "mounted" into an internal catalog.

Issue Links

• is related to: SPARK-23443 Spark with Glue as external catalog
• relates to: SPARK-15691 Refactor and improve Hive support
• relates to: SPARK-24814 Relationship between catalog and datasources
• relates to: SPARK-24017 Refactor ExternalCatalog to be an interface

Sub-Tasks

• Spark SQL ExternalCatalog API custom implementation support: Closed