[SPARK-2045] Sort-based shuffle implementation

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Shuffle, Spark Core
• Target Version/s: 1.1.0
• Assignee: Matei Alexandru Zaharia
• Reporter: Matei Alexandru Zaharia
• Created: 05/Jun/14 23:15
• Updated: 23/Sep/14 02:52
• Resolved: 31/Jul/14 05:49

Description

Building on the pluggability in SPARK-2044, a sort-based shuffle implementation that takes advantage of an Ordering for keys (or just sorts by hashcode for keys that don't have it) would likely improve performance and memory usage in very large shuffles. Our current hash-based shuffle needs an open file for each reduce task, which can fill up a lot of memory for compression buffers and cause inefficient IO. This would avoid both of those issues.

Issue Links

• depends upon: SPARK-2044 Pluggable interface for shuffles
• is depended upon by: SPARK-2213 Sort Merge Join
• relates to: SPARK-3655 Support sorting of values in addition to keys (i.e. secondary sort)
• links to: [Github] Pull Request #1499 (mateiz)