[SPARK-8160] Tungsten style external aggregation

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Epic Link: Tungsten Phase 1
• Sprint: Spark 1.5 release
• Assignee: Yin Huai
• Reporter: Reynold Xin
• Created: 08/Jun/15 07:59
• Updated: 07/Aug/15 23:27
• Resolved: 07/Aug/15 23:27

Description

Support using external sorting to run aggregate so we can easily process aggregates where each partition is much larger than memory size.

Issue Links

• links to: [Github] Pull Request #6875 (lianhuiwang)
• links to: [Github] Pull Request #7423 (lianhuiwang)
• links to: [Github] Pull Request #7827 (lianhuiwang)

Sub-Tasks

• BytesToBytesMap should encode data the same way as UnsafeExternalSorter: Resolved
• UnsafeFixedWidthAggregationMap should support in-place sorting of its own records: Resolved
• UnsafeFixedWidthAggregationMap should be able to turn itself into an UnsafeKVExternalSorter: Resolved
• Add randomized testing for UnsafeKVExternalSorter: Resolved
• Surface concrete iterator types in various sort classes: Resolved
• UnsafeFixedWidthAggregationMap.destructAndCreateExternalSorter will add an empty entry to if the map is empty.: Resolved