[SPARK-3018] Release all ManagedBuffers upon task completion/failure

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Shuffle, Spark Core
• Target Version/s: 1.2.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 13/Aug/14 22:38
• Updated: 04/Nov/14 08:40
• Resolved: 04/Nov/14 08:40

Description

BlockFetcherIterator retains ManagedBuffers returned by BlockClient.fetchBlocks. Those buffers are released when the iterators are traversed fully. In the case of task failures or completion without exhausting the iterator, this could lead to memory leak.

Issue Links

• links to: Github pull request