[HIVE-12880] spark-assembly causes Hive class version problems

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Assignee: Sergey Shelukhin Assign to me
• Reporter: Hui Zheng
• Created: 15/Jan/16 20:28
• Updated: 21/Jun/16 15:53
• Resolved: 20/Feb/16 22:58

Description

It looks like spark-assembly contains versions of Hive classes (e.g. HiveConf), and these sometimes (always?) come from older versions of Hive.
We've seen problems where depending on classpath perturbations, NoSuchField errors may be thrown for recently added ConfVars because the HiveConf class comes from spark-assembly.

Would making sure spark-assembly comes last in the classpath solve the problem?
Otherwise, can we depend on something that does not package older Hive classes?

Currently, HIVE-12179 provides a workaround (in non-Spark use case, at least; I am assuming this issue can also affect Hive-on-Spark).

Issue Links

• is related to: HIVE-12179 Add option to not add spark-assembly.jar to Hive classpath
• is related to: HIVE-9608 Define SPARK_HOME if not defined automagically