[HIVE-14029] Update Spark version to 2.0.0

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Labels: Incompatible TODOC2.2
• Assignee: Ferdinand Xu
• Reporter: Ferdinand Xu
• Created: 16/Jun/16 02:11
• Updated: 21/Jul/17 17:55
• Resolved: 28/Sep/16 01:22

Description

There are quite some new optimizations in Spark 2.0.0. We need to bump up Spark to 2.0.0 to benefit those performance improvements.
To update Spark version to 2.0.0, the following changes are required:

• Spark API updates:
  • SparkShuffler#call return Iterator instead of Iterable
  • SparkListener -> JavaSparkListener
  • InputMetrics constructor doesn’t accept readMethod
  • Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics return long type instead of integer

• Dependency upgrade:
  • Jackson: 2.4.2 -> 2.6.5
  • Netty version: 4.0.23.Final -> 4.0.29.Final
  • Scala binary version: 2.10 -> 2.11
  • Scala version: 2.10.4 -> 2.11.8

Issue Links

• is duplicated by: HIVE-14777 Add support of Spark-2.0.0 in Hive-2.X.X
• is related to: SPARK-12154 Upgrade to Jersey 2
• is related to: SPARK-17563 Add org/apache/spark/JavaSparkListener to make Spark-2.0.0 work with Hive-2.X.X
• is related to: HIVE-14825 Figure out the minimum set of required jars for Hive on Spark after bumping up to Spark 2.0.0
• relates to: HIVE-14919 Improve the performance of Hive on Spark 2.0.0
• links to: GitHub Pull Request #103
• links to: review board