[MAPREDUCE-5493] In-memory map outputs can be leaked after shuffle completes

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.1.0-beta, 0.23.9
• Fix Version/s: 2.1.1-beta
• Component/s: mrv2
• Target Version/s: 0.23.10, 2.1.1-beta
• Assignee: Jason Darrell Lowe
• Reporter: Jason Darrell Lowe
• Created: 03/Sep/13 21:51
• Updated: 19/Mar/14 18:42
• Resolved: 16/Sep/13 14:20

Description

MergeManagerImpl#close adds the contents of inMemoryMergedMapOutputs and inMemoryMapOutputs to a list of map outputs that is subsequently processed, but it does not clear those sets. This prevents some of the map outputs from being garbage collected and significantly reduces the memory available for the subsequent reduce phase.

Issue Links

• is cloned by: MAPREDUCE-5543 In-memory map outputs can be leaked after shuffle completes in 0.23
• relates to: TEZ-953 Port MAPREDUCE-5493