[MAPREDUCE-6334] Fetcher#copyMapOutput is leaking usedMemory upon IOException during InMemoryMapOutput shuffle handler

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.8.0, 2.7.1, 2.6.2, 3.0.0-alpha1
• Target Version/s: 2.7.1, 2.6.2
• Hadoop Flags: Reviewed
• Assignee: Eric Payne
• Reporter: Eric Payne
• Created: 23/Apr/15 20:22
• Updated: 05/Jan/17 22:38
• Resolved: 28/Apr/15 20:21

Description

We are seeing this happen when

• an NM's disk goes bad during the creation of map output(s)
• the reducer's fetcher can read the shuffle header and reserve the memory
• but gets an IOException when trying to shuffle for InMemoryMapOutput
• shuffle fetch retry is enabled

Issue Links

• is duplicated by: MAPREDUCE-6351 Reducer hung in copy phase.