[SPARK-23346] Failed tasks reported as success if the failure reason is not ExceptionFailure

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.2.0
• Component/s: Spark Core, SQL
• Labels: bulk-closed

Environment

HADOOP 2.6 + JDK1.8 + SPARK 2.2.0

• Assignee: Unassigned
• Reporter: 吴志龙
• Created: 07/Feb/18 02:40
• Updated: 21/May/19 04:15
• Resolved: 21/May/19 04:15

Description

We have many other failure reasons, such as TaskResultLost,but the status is success. In the web ui, we count non-ExceptionFailure failures as successful tasks, which is highly misleading.

detail message:
Job aborted due to stage failure: Task 0 in stage 7.0 failed 10 times, most recent failure: Lost task 0.9 in stage 7.0 (TID 35, 60.hadoop.com, executor 27): TaskResultLost (result lost from block manager)