[SPARK-1272] Don't fail job if some local directories are buggy

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: Shuffle, Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Patrick Wendell
• Created: 18/Mar/14 18:02
• Updated: 17/May/20 18:30
• Resolved: 21/May/19 05:36

Description

If Spark cannot create shuffle directories inside of a local directory it might make sense to just log an error and continue, provided that at least one valid shuffle directory exists. Otherwise if a single disk is wonky the entire job can fail.

The down side is that this might mask failures if the person actually misconfigures the local directories to point to the wrong disk(s).