[SPARK-4188] Shuffle fetches should be retried at a lower level

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.2.0
• Component/s: Spark Core
• Assignee: Aaron Davidson
• Reporter: Aaron Davidson
• Created: 01/Nov/14 21:46
• Updated: 07/Nov/14 02:40
• Resolved: 07/Nov/14 02:40

Description

During periods of high network (or GC) load, it is not uncommon that IOExceptions crop up around connection failures when fetching shuffle files. Unfortunately, when such a failure occurs, it is interpreted as an inability to fetch the files, which causes us to mark the executor as lost and recompute all of its shuffle outputs.
We should allow retrying at the network level in the event of an IOException in order to avoid this circumstance.

Issue Links

• is duplicated by: SPARK-4238 Perform network-level retry of shuffle file fetches
• is duplicated by: SPARK-4195 retry to fetch blocks's result when fetchfailed's reason is connection timeout
• links to: [Github] Pull Request #3101 (aarondav)