[SPARK-7855] Move hash-style shuffle code out of ExternalSorter and into own file

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: Shuffle, Spark Core
• Target Version/s: 1.5.0
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 25/May/15 17:43
• Updated: 17/May/20 18:30
• Resolved: 30/May/15 22:27

Description

ExternalSorter contains a bunch of code for handling the bypassMergeThreshold / hash-style shuffle path. I think that it would significantly simplify the code to move this functionality out of ExternalSorter and into a separate class which shares a common interface (insertAll / writePartitionedFile()). This is a stepping-stone towards eventually removing this bypass path (see SPARK-6026)

Issue Links

• blocks: SPARK-6026 Eliminate the bypassMergeThreshold parameter and associated hash-ish shuffle within the Sort shuffle code
• links to: [Github] Pull Request #6397 (JoshRosen)