[SPARK-17769] Some FetchFailure refactoring in the DAGScheduler

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: Scheduler, Spark Core
• Assignee: Mark Hamstra
• Reporter: Mark Hamstra
• Created: 03/Oct/16 18:51
• Updated: 17/May/20 17:47
• Resolved: 16/Dec/16 20:47

Description

SPARK-17644 opened up a discussion about further refactoring of the DAGScheduler's handling of FetchFailure events. These include:

• rewriting code and comments to improve readability
• doing fetchFailedAttemptIds.add(stageAttemptId) even when disallowStageRetryForTest is true
• issuing a ResubmitFailedStages event based on whether one is already enqueued for the current failed stage, not any prior failed stage
• logging the resubmission of all failed stages

Issue Links

• links to: [Github] Pull Request #15335 (markhamstra)