[SPARK-23253] Only write shuffle temporary index file when there is not an existing one

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.1
• Fix Version/s: 2.4.0
• Component/s: Shuffle, Spark Core
• Assignee: Kent Yao 2
• Reporter: Kent Yao 2
• Created: 29/Jan/18 10:11
• Updated: 03/Sep/18 02:41
• Resolved: 02/Feb/18 15:11

Description

Shuffle Index temporay file is used for atomic creating shuffle index file, it is not needed when the index file already exists after another attempts of same task had it done.

Issue Links

• links to: [Github] Pull Request #20422 (yaooqinn)