[SPARK-14269] Eliminate unnecessary submitStage() call.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Component/s: Scheduler, Spark Core
• Assignee: Takuya Ueshin
• Reporter: Takuya Ueshin
• Created: 30/Mar/16 09:28
• Updated: 17/May/20 17:48
• Resolved: 25/May/16 20:58

Description

Currently a method submitStage() for waiting stages is called on every iteration of the event loop in DAGScheduler to submit all waiting stages, but most of them are not necessary because they are not related to Stage status.
The case we should try to submit waiting stages is only when their parent stages are successfully completed.

This elimination can improve DAGScheduler performance.

Issue Links

• relates to: SPARK-13902 Make DAGScheduler not to create duplicate stage.
• links to: [Github] Pull Request #12060 (ueshin)