[HIVE-7893] Find a way to get a job identifier when submitting a spark job [Spark Branch]

• Type: Task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Component/s: Spark
• Labels: Spark-M3
• Assignee: Rui Li
• Reporter: Rui Li
• Created: 27/Aug/14 14:54
• Updated: 20/Oct/14 05:06
• Resolved: 20/Oct/14 02:10

Description

Currently we use the foreach RDD action to submit a spark job. In order to implement job monitoring functionality (HIVE-7438), we need to get a job identifier when submitting the job, so that we can later register some listener for that specific job.

This task requires facilitation from spark side (SPARK-2636). I've tried to use AsyncRDDActions instead of the traditional actions, and it proved to be a possible way to get the job ID we need.

Issue Links

• depends upon: SPARK-2636 Expose job ID in JobWaiter API
• is part of: HIVE-7438 Counters, statistics, and metrics [Spark Branch]
• is part of: HIVE-7292 Hive on Spark
• is related to: HIVE-7439 Spark job monitoring and error reporting [Spark Branch]