[SPARK-2636] Expose job ID in JobWaiter API

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Java API
• Labels: hive
• Target Version/s: 1.2.0
• Assignee: Chengxiang Li
• Reporter: Chengxiang Li
• Created: 23/Jul/14 03:01
• Updated: 02/Sep/14 06:28
• Resolved: 02/Sep/14 06:28

Description

In Hive on Spark, we want to track spark job status through Spark API, the basic idea is as following:

1. create an hive-specified spark listener and register it to spark listener bus.
2. hive-specified spark listener generate job status by spark listener events.
3. hive driver track job status through hive-specified spark listener.

the current problem is that hive driver need job identifier to track specified job status through spark listener, but there is no spark API to get job identifier(like job id) while submit spark job.

I think other project whoever try to track job status with spark API would suffer from this as well.

Issue Links

• is depended upon by: SPARK-3145 Hive on Spark umbrella
• is depended upon by: HIVE-7552 Collect spark job statistic through spark metrics [Spark Branch]
• is depended upon by: HIVE-7292 Hive on Spark
• is depended upon by: HIVE-7893 Find a way to get a job identifier when submitting a spark job [Spark Branch]
• relates to: SPARK-2321 Design a proper progress reporting & event listener API
• links to: [Github] Pull Request #2176 (lirui-intel)