[SPARK-7542] Support off-heap sort buffer in UnsafeExternalSorter

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.6.0
• Component/s: Spark Core
• Epic Link: Tungsten Phase 2
• Assignee: Davies Liu
• Reporter: Josh Rosen
• Created: 11/May/15 21:46
• Updated: 06/Nov/15 03:02
• Resolved: 06/Nov/15 03:02

Description

UnsafeExternalSorter, introduced in SPARK-7081, uses on-heap long[] arrays as its sort buffers. When records are small, the sorting array might be as large as the data pages, so it would be useful to be able to allocate this array off-heap (using our unsafe LongArray). Unfortunately, we can't currently do this because TimSort calls allocate() to create data buffers but doesn't call any corresponding cleanup methods to free them.

We should look into extending TimSort with buffer freeing methods, then consider switching to LongArray in UnsafeShuffleSortDataFormat.

Issue Links

• is blocked by: SPARK-7081 Faster sort-based shuffle path using binary processing cache-aware sort
• links to: [Github] Pull Request #8068 (viirya)
• links to: [Github] Pull Request #9477 (davies)