[SPARK-2420] Dependency changes for compatibility with Hive

• Type: Wish
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.2.0
• Component/s: Build
• Labels: Hive
• Assignee: Unassigned
• Reporter: Xuefu Zhang
• Created: 09/Jul/14 19:44
• Updated: 01/Nov/14 13:54
• Resolved: 20/Aug/14 23:25

Description

During the prototyping of HIVE-7292, many library conflicts showed up because Spark build contains versions of libraries that's vastly different from current major Hadoop version. It would be nice if we can choose versions that's in line with Hadoop or shading them in the assembly. Here are the wish list:

1. Upgrade protobuf version to 2.5.0 from current 2.4.1
2. Shading Spark's jetty and servlet dependency in the assembly.
3. guava version difference. Spark is using a higher version. I'm not sure what's the best solution for this.

The list may grow as HIVE-7292 proceeds.

For information only, the attached is a patch that we applied on Spark in order to make Spark work with Hive. It gives an idea of the scope of changes.

Issue Links

• is depended upon by: HIVE-7292 Hive on Spark
• is related to: HIVE-7437 Check if servlet-api and jetty module in Spark library are an issue for hive-spark integration [Spark Branch]
• is required by: HIVE-7387 Guava version conflict between hadoop and spark [Spark-Branch]
• links to: [Github] Pull Request #1610 (srowen)

Sub-Tasks

• Shade Guava in Spark deliverables: Resolved