[SPARK-11293] ExternalSorter and ExternalAppendOnlyMap should free shuffle memory in their stop() methods

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.3.1, 1.4.1, 1.5.1, 1.6.0
• Fix Version/s: 1.6.0
• Component/s: Spark Core
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 24/Oct/15 00:29
• Updated: 09/Apr/19 14:53
• Resolved: 31/May/16 21:59

Description

I discovered multiple leaks of shuffle memory while working on my memory manager consolidation patch, which added the ability to do strict memory leak detection for the bookkeeping that used to be performed by the ShuffleMemoryManager. This uncovered a handful of places where tasks can acquire execution/shuffle memory but never release it, starving themselves of memory.

Problems that I found:

• ExternalSorter.stop() should release the sorter's shuffle/execution memory.
• BlockStoreShuffleReader should call ExternalSorter.stop() using a CompletionIterator.
• ExternalAppendOnlyMap exposes no equivalent of stop() for freeing its resources.

Issue Links

• is cloned by: SPARK-27417 CLONE - ExternalSorter and ExternalAppendOnlyMap should free shuffle memory in their stop() methods
• is related to: SPARK-14560 Cooperative Memory Management for Spillables
• is related to: SPARK-14168 Managed Memory Leak Msg Should Only Be a Warning
• links to: [Github] Pull Request #9260 (JoshRosen)
• links to: [Github] Pull Request #9427 (JoshRosen)
• links to: [Github] Pull Request #13027 (lianhuiwang)