[SPARK-2321] Design a proper progress reporting & event listener API

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.2.0
• Component/s: Java API, Spark Core
• Target Version/s: 1.2.0
• Epic Link: Monitoring and debugging improvements (Spark 1.2)
• Assignee: Josh Rosen
• Reporter: Reynold Xin
• Created: 29/Jun/14 08:42
• Updated: 03/Dec/14 20:52
• Resolved: 25/Oct/14 07:07

Description

This is a ticket to track progress on redesigning the SparkListener and JobProgressListener API.

There are multiple problems with the current design, including:

0. I'm not sure if the API is usable in Java (there are at least some enums we used in Scala and a bunch of case classes that might complicate things).
1. The whole API is marked as DeveloperApi, because we haven't paid a lot of attention to it yet. Something as important as progress reporting deserves a more stable API.
2. There is no easy way to connect jobs with stages. Similarly, there is no easy way to connect job groups with jobs / stages.
3. JobProgressListener itself has no encapsulation at all. States can be arbitrarily mutated by external programs. Variable names are sort of randomly decided and inconsistent.

We should just revisit these and propose a new, concrete design.

Issue Links

• blocks: SPARK-4017 Progress bar in console
• is duplicated by: SPARK-2633 enhance spark listener API to gather more spark job information
• is related to: SPARK-2633 enhance spark listener API to gather more spark job information
• is related to: SPARK-2636 Expose job ID in JobWaiter API
• is related to: SPARK-576 Design and develop a more precise progress estimator
• is related to: SPARK-3059 Spark internal module interface design
• relates to: SPARK-3644 REST API for Spark application info (jobs / stages / tasks / storage info)
• relates to: SPARK-4440 Enhance the job progress API to expose more information
• relates to: SPARK-4567 Make SparkJobInfo and SparkStageInfo serializable
• links to: [Github] Pull Request #2696 (JoshRosen)
• links to: [Github] Pull Request #3197 (JoshRosen)