[SPARK-21492] Memory leak in SortMergeJoin

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0, 2.3.0, 2.3.1, 3.0.0
• Fix Version/s: 2.4.5, 3.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Yuanjian Li
• Reporter: Zhan Zhang
• Created: 20/Jul/17 21:34
• Updated: 04/Jan/24 00:31
• Resolved: 22/Oct/19 11:09

Description

In SortMergeJoin, if the iterator is not exhausted, there will be memory leak caused by the Sort. The memory is not released until the task end, and cannot be used by other operators causing performance drop or OOM.

Issue Links

• blocks: SPARK-27558 NPE in TaskCompletionListener due to Spark OOM in UnsafeExternalSorter causing tasks to hang
• is duplicated by: SPARK-24657 SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join
• links to: [Github] Pull Request #18694 (zhzhan)
• links to: GitHub Pull Request #18694
• links to: GitHub Pull Request #23762
• links to: GitHub Pull Request #25888
• links to: GitHub Pull Request #26164
• links to: GitHub Pull Request #26210
• links to: GitHub Pull Request #26229
• links to: GitHub Pull Request #44573