[SPARK-14915] Tasks that fail due to CommitDeniedException (a side-effect of speculation) can cause job to never complete

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.5.3, 1.6.2, 2.0.0
• Fix Version/s: 1.6.2, 2.0.0
• Target Version/s: 2.0.0
• Assignee: Jason Moore
• Reporter: Jason Moore
• Created: 26/Apr/16 07:47
• Updated: 05/Aug/16 21:13
• Resolved: 05/May/16 10:02

Description

In SPARK-14357, code was corrected towards the originally intended behavior that a CommitDeniedException should not count towards the failure count for a job. After having run with this fix for a few weeks, it's become apparent that this behavior has some unintended consequences - that a speculative task will continuously receive a CDE from the driver, now causing it to fail and retry over and over without limit.

I'm thinking we could put a task that receives a CDE from the driver, into a TaskState.FINISHED or some other state to indicated that the task shouldn't be resubmitted by the TaskScheduler. I'd probably need some opinions on whether there are other consequences for doing something like this.

Issue Links

• is duplicated by: SPARK-16709 Task with commit failed will retry infinite when speculation set to true
• is related to: SPARK-14357 Tasks that fail due to CommitDeniedException (a side-effect of speculation) can cause job failure
• links to: [Github] Pull Request #12751 (jasonmoore2k)
• links to: [Github] Pull Request #12950 (srowen)