[SPARK-29657] Iterator spill supporting radix sort with null prefix

• Type: Bug
• Status: Resolved
• Priority: Trivial
• Resolution: Incomplete
• Affects Version/s: 2.4.0
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: dzcxzl
• Created: 30/Oct/19 11:12
• Updated: 07/Jul/21 10:44
• Resolved: 25/May/21 01:42

Description

In the case of radix sort, when the insertRecord part of the keyPrefix is null, the iterator type returned by getSortedIterator is ChainedIterator.
Currently ChainedIterator does not support spill, causing UnsafeExternalSorter to take up a lot of execution memory, allocatePage fails, throw SparkOutOfMemoryError Unable to acquire xxx bytes of memory, got 0

The following is a log of an error we encountered in the production environment.

[Executor task launch worker for task 66055] INFO TaskMemoryManager: Memory used in task 66055
[Executor task launch worker for task 66055] INFO TaskMemoryManager: Acquired by org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@39dd866e: 64.0 KB
[Executor task launch worker for task 66055] INFO TaskMemoryManager: Acquired by org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@74d17927: 4.6 GB
[Executor task launch worker for task 66055] INFO TaskMemoryManager: Acquired by org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@31478f9c: 61.0 MB
[Executor task launch worker for task 66055] INFO TaskMemoryManager: 0 bytes of memory were used by task 66055 but are not associated with specific consumers
[Executor task launch worker for task 66055] INFO TaskMemoryManager: 4962998749 bytes of memory are used for execution and 2218326 bytes of memory are used for storage
[Executor task launch worker for task 66055] ERROR Executor: Exception in task 42.3 in stage 29.0 (TID 66055)
SparkOutOfMemoryError: Unable to acquire 3436 bytes of memory, got 0

Issue Links

• is fixed by: SPARK-32900 UnsafeExternalSorter.SpillableIterator cannot spill when there are NULLs in the input and radix sorting is used.
• links to: GitHub Pull Request #26323