[SPARK-17767] Spark SQL ExternalCatalog API custom implementation support

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Later
• Affects Version/s: 2.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Alex Liu
• Created: 03/Oct/16 15:19
• Updated: 24/Jan/20 04:10
• Resolved: 03/Oct/16 23:22

Description

There is no way/easy way to configure Spark to use customized ExternalCatalog. Internal source code is hardcoded to use either hive or in-memory metastore. Spark SQL thriftserver is hardcoded to use HiveExternalCatalog. We should be able to create a custom external catalog and thriftserver should be able to use it. Potentially Spark SQL thriftserver shouldn't depend on Hive thriftserer.

Issue Links

• is duplicated by: SPARK-19632 Allow configuring non-hive and non-local SessionState and ExternalCatalog
• is duplicated by: SPARK-30617 Is there any possible that spark no longer restrict enumerate types of spark.sql.catalogImplementation
• is related to: SPARK-23443 Spark with Glue as external catalog
• is required by: IGNITE-3084 Spark Data Frames Support in Apache Ignite
• links to: [Github] Pull Request #15336 (dongjoon-hyun)