[SPARK-7829] SortShuffleWriter writes inconsistent data & index files on stage retry

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.1
• Fix Version/s: 1.5.3, 1.6.0
• Component/s: Shuffle, Spark Core
• Target Version/s: 1.5.3, 1.6.0
• Assignee: Davies Liu
• Reporter: Imran Rashid
• Created: 22/May/15 14:46
• Updated: 13/Nov/15 21:57
• Resolved: 13/Nov/15 21:57

Description

When a stage is retried, even if a shuffle map task was successful, it may get retried in any case. If it happens to get scheduled on the same executor, the old data file is appended, while the index file still assumes the data starts in position 0. This leads to an apparently corrupt shuffle map output, since when the data file is read, the index file points to the wrong location.

Issue Links

• is related to: SPARK-7308 Should there be multiple concurrent attempts for one stage?
• relates to: SPARK-8029 ShuffleMapTasks must be robust to concurrent attempts on the same executor
• links to: [Github] Pull Request #6420 (squito)