[SPARK-10781] Allow certain number of failed tasks and allow job to succeed

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.5.0
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Thomas Graves
• Created: 23/Sep/15 19:52
• Updated: 25/May/21 01:53
• Resolved: 25/May/21 01:42

Description

MapReduce has this config mapreduce.map.failures.maxpercent and mapreduce.reduce.failures.maxpercent which allows for a certain percent of tasks to fail but the job to still succeed.

This could be a useful feature in Spark also if a job doesn't need all the tasks to be successful.