[HIVE-8548] Integrate with remote Spark context after HIVE-8528 [Spark Branch]

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Spark
• Assignee: Chengxiang Li
• Reporter: Xuefu Zhang
• Created: 21/Oct/14 22:47
• Updated: 19/Dec/14 07:01
• Resolved: 19/Dec/14 07:01

Description

With HIVE-8528, HiverSever2 should use remote Spark context to submit job and monitor progress, etc. This is necessary if Hive runs on standalone cluster, Yarn, or Mesos. If Hive runs with spark.master=local, we should continue using SparkContext in current way.
We take this as root JIRA to track all Remote Spark Context integration related subtasks.

Issue Links

• depends upon: HIVE-8528 Add remote Spark client to Hive [Spark Branch]
• is part of: HIVE-7292 Hive on Spark

Sub-Tasks

• Unify spark client API and implement remote spark client.[Spark Branch]: Resolved
• enable job progress monitoring of Remote Spark Context [Spark Branch]: Resolved
• Enhance metrics gathering in Spark Client [Spark Branch]: Resolved
• identify dependency scope for Remote Spark Context.[Spark Branch]: Resolved
• Enable automatic tests with remote spark client [Spark Branch]: Resolved
• Guava dependency conflict between hive driver and remote spark context[Spark Branch]: Resolved
• Automatic calculate reduce number for spark job [Spark Branch]: Resolved
• SparkSession and SparkClient mapping[Spark Branch]: Resolved
• Spark remote context doesn't work with local-cluster [Spark Branch]: Resolved
• Hive hangs while some error/exception happens beyond job execution [Spark Branch]: Resolved
• Remote spark context needs to clean up itself in case of connection timeout [Spark Branch]: Open
• SparkSession is not closed until JVM exit.[Spark Branch]: Resolved
• Implement more fine-grained remote client-level events [Spark Branch]: Resolved
• Enable job metrics/statistics gathering for remote spark context [Spark Branch]: Resolved
• Fix custom_input_output_format [Spark Branch]: Resolved