[SPARK-32900] UnsafeExternalSorter.SpillableIterator cannot spill when there are NULLs in the input and radix sorting is used.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.7, 3.0.1
• Fix Version/s: 2.4.8, 3.0.2, 3.1.0
• Component/s: Spark Core
• Assignee: Tom van Bussel
• Reporter: Tom van Bussel
• Created: 16/Sep/20 09:19
• Updated: 07/Jul/21 10:44
• Resolved: 17/Sep/20 10:38

Description

In order to determine whether UnsafeExternalSorter.SpillableIterator has spilled already it checks whether upstream is an instance of UnsafeInMemorySorter.SortedIterator. When radix sorting is used (added by SPARK-14851) and there are NULLs in the input however, upstream will be an instance of UnsafeExternalSorter.ChainedIterator instead, but should still be spilled.

Issue Links

• fixes: SPARK-29657 Iterator spill supporting radix sort with null prefix
• links to: [Github] Pull Request #29772 (tomvanbussel)