[SPARK-9808] Remove hash shuffle file consolidation

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: Shuffle, Spark Core
• Target Version/s: 1.6.0
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 11/Aug/15 01:32
• Updated: 18/Sep/15 20:49
• Resolved: 18/Sep/15 20:48

Description

I think that we should remove spark.shuffle.consolidateFiles and its associated implementation for Spark 1.5.0. This feature isn't properly tested and does not work with the external shuffle service. Since it's likely to be buggy and since its motivation has been subsumed by sort-based shuffle, I think it's a prime candidate for removal now.

Issue Links

• links to: [Github] Pull Request #8089 (JoshRosen)
• links to: [Github] Pull Request #8812 (rxin)