[SPARK-32911] UnsafeExternalSorter.SpillableIterator cannot spill after reading all records

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.7, 3.0.1
• Fix Version/s: 3.1.0, 3.0.4
• Component/s: Spark Core
• Assignee: Tom van Bussel
• Reporter: Tom van Bussel
• Created: 17/Sep/20 12:06
• Updated: 31/May/22 22:57
• Resolved: 18/Sep/20 11:49

Description

No memory is freed after calling UnsafeExternalSorter.SpillableIterator.spill() when all records have been read, even though it is still holding onto some memory. This may starve other {{MemoryConsumer}}s of memory.

Issue Links

• links to: [Github] Pull Request #29787 (tomvanbussel)