[SPARK-8029] ShuffleMapTasks must be robust to concurrent attempts on the same executor

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.5.3, 1.6.0
• Component/s: Spark Core
• Target Version/s: 1.5.3, 1.6.0
• Assignee: Davies Liu
• Reporter: Imran Rashid
• Created: 02/Jun/15 05:17
• Updated: 09/Jan/17 20:54
• Resolved: 13/Nov/15 06:45

Description

When stages get retried, a task may have more than one attempt running at the same time, on the same executor. Currently this causes problems for ShuffleMapTasks, since all attempts try to write to the same output files.

This is finally resolved through https://github.com/apache/spark/pull/9610, which uses the first writer wins approach.

Issue Links

• is related to: SPARK-7829 SortShuffleWriter writes inconsistent data & index files on stage retry
• is related to: SPARK-18113 Sending AskPermissionToCommitOutput failed, driver enter into task deadloop
• is required by: SPARK-7308 Should there be multiple concurrent attempts for one stage?
• links to: [Github] Pull Request #6648 (squito)
• links to: [Github] Pull Request #9214 (squito)
• links to: [Github] Pull Request #9610 (davies)
• links to: [Github] Pull Request #9686 (davies)