[SPARK-19631] OutputCommitCoordinator should not allow commits for already failed tasks

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.0
• Component/s: Scheduler, Spark Core
• Assignee: Patrick Woody
• Reporter: Patrick Woody
• Created: 16/Feb/17 16:44
• Updated: 17/May/20 17:48
• Resolved: 02/Mar/17 23:57

Description

This is similar to SPARK-6614, but there a race condition where a task may fail (e.g. Executor heartbeat timeout) and still manage to go through the commit protocol successfully. After this any retries of the task will fail indefinitely because of TaskCommitDenied.

Issue Links

• relates to: SPARK-19790 OutputCommitCoordinator should not allow another task to commit after an ExecutorFailure
• links to: [Github] Pull Request #16959 (pwoody)