[SPARK-26614] Speculation kill might cause job failure

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.2, 2.3.1, 2.4.0
• Component/s: Scheduler, Spark Core
• Assignee: Unassigned
• Reporter: liupengcheng
• Created: 14/Jan/19 11:22
• Updated: 16/Jan/19 02:46
• Resolved: 16/Jan/19 02:44

Description

This issue is similar to SPARK-26612

Some odd exceptions might be thrown in speculation kill, however, currently spark does not handle this case and will report failure to Driver. This exception will be counting towards MAX_TASK_FAILURES, and might result in the job failure.

 

I think we can check state of task to tell if we report failure or killed.

Issue Links

• links to: GitHub Pull Request #23537