[SPARK-30617] Is there any possible that spark no longer restrict enumerate types of spark.sql.catalogImplementation

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 3.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: weiwenda
• Created: 23/Jan/20 09:25
• Updated: 25/Jan/20 04:35
• Resolved: 25/Jan/20 04:34

Description

1. We have implemented a complex ExternalCatalog which is used for retrieving multi isomerism database's metadata(sush as elasticsearch、postgresql), so that we can make a mixture query between hive and our online data.
2. But as spark require that value of spark.sql.catalogImplementation must be one of in-memory/hive, we have to modify SparkSession and rebuild spark to make our project work.
3. Finally, we hope spark removing above restriction, so that it's will be much easier to let us keep pace with new spark version. Thanks!

Issue Links

• duplicates: SPARK-17767 Spark SQL ExternalCatalog API custom implementation support
• duplicates: SPARK-19632 Allow configuring non-hive and non-local SessionState and ExternalCatalog
• is superceded by: SPARK-24252 DataSourceV2: Add catalog support
• relates to: SPARK-23443 Spark with Glue as external catalog
• links to: GitHub Pull Request #27338
• links to: GitHub Pull Request #27349