[SPARK-6906] Improve Hive integration support

• Type: Story
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Sprint: Spark 1.5 release
• Assignee: Michael Armbrust
• Reporter: Michael Armbrust
• Created: 14/Apr/15 18:09
• Updated: 27/Aug/15 13:02
• Resolved: 05/Aug/15 19:35

Description

Right now Spark SQL is very coupled to a specific version of Hive for two primary reasons.

• Metadata: we use the Hive Metastore client to retrieve information about tables in a metastore.
• Execution: UDFs, UDAFs, SerDes, HiveConf and various helper functions for configuration.

Since Hive is generally not compatible across versions, we are currently maintain fairly expensive shim layers to let us talk to both Hive 12 and Hive 13 metastores. Ideally we would be able to talk to more versions of Hive with less maintenance burden.

This task is proposing that we separate the hive version that is used for communicating with the metastore from the version that is used for execution. In doing so we can significantly reduce the size of the shim by only providing compatibility for metadata operations. All execution will be done with single version of Hive (the newest version that is supported by Spark SQL).

Issue Links

• links to: [Github] Pull Request #6167 (marmbrus)