[HIVE-7439] Spark job monitoring and error reporting [Spark Branch]

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Spark
• Labels: Spark-M3 TODOC-SPARK TODOC15
• Assignee: Chengxiang Li
• Reporter: Xuefu Zhang
• Created: 17/Jul/14 21:00
• Updated: 29/May/15 08:18
• Resolved: 14/Oct/14 02:57

Description

After Hive submits a job to Spark cluster, we need to report to user the job progress, such as the percentage done, to the user. This is especially important for long running queries. Moreover, if there is an error during job submission or execution, it's also crucial for hive to fetch the error log and/or stacktrace and feedback it to the user.

Please refer design doc on wiki for more information.

CLEAR LIBRARY CACHE

Issue Links

• is related to: HIVE-8834 enable job progress monitoring of Remote Spark Context [Spark Branch]
• relates to: HIVE-7874 Support multiple concurrent users
• relates to: SPARK-3902 Stabilize AsyncRDDActions and expose its methods in Java API
• relates to: HIVE-7893 Find a way to get a job identifier when submitting a spark job [Spark Branch]
• links to: https://reviews.apache.org/r/26545/