[SPARK-26525] Fast release memory of ShuffleBlockFetcherIterator

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.2
• Fix Version/s: 3.0.0
• Component/s: Shuffle, Spark Core
• Assignee: liupengcheng
• Reporter: liupengcheng
• Created: 03/Jan/19 09:24
• Updated: 17/May/20 18:31
• Resolved: 11/Feb/19 04:29

Description

Currently, spark would not release ShuffleBlockFetcherIterator until the whole task finished.

In some conditions, it incurs memory leak.

An example is Shuffle -> map -> Coalesce(shuffle = false). Each ShuffleBlockFetcherIterator contains  some metas about MapStatus(blocksByAddress) and each ShuffleMapTask will keep n(max to shuffle partitions) shuffleBlockFetcherIterator for they are refered by onCompleteCallbacks of TaskContext, in some case, it may take huge memory and the memory will not released until the task finished.

Actually, We can release ShuffleBlockFetcherIterator as soon as it's consumed.

Issue Links

• links to: GitHub Pull Request #23438