[SPARK-10783] Do track the pointer array in UnsafeInMemorySorter

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.5.1, 1.6.0
• Fix Version/s: 1.6.0
• Component/s: SQL
• Target Version/s: 1.6.0
• Assignee: Davies Liu
• Reporter: Andrew Or
• Created: 23/Sep/15 21:44
• Updated: 09/Nov/15 18:06
• Resolved: 09/Nov/15 18:06

Description

SPARK-10474 (https://github.com/apache/spark/pull/8888) removed the pointer array tracking because `TungstenAggregate` would fail under memory pressure. However, this is somewhat of a hack that we should fix in the right way in 1.6.0 to ensure we don't OOM because of this.

Issue Links

• is related to: SPARK-10474 TungstenAggregation cannot acquire memory for pointer array after switching to sort-based
• relates to: SPARK-10929 Tungsten fails to acquire memory writing to HDFS