[MAPREDUCE-1182] Reducers fail with OutOfMemoryError while copying Map outputs

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.20.2
• Hadoop Flags: Reviewed
• Release Note: Modifies shuffle related memory parameters to use 'long' from 'int' so that sizes greater than maximum integer size are handled correctly
• Tags: OutOfMemoryError, OOM reducer
• Assignee: Chandra Prakash Bhagtani
• Reporter: Chandra Prakash Bhagtani
• Created: 03/Nov/09 10:06
• Updated: 21/Dec/09 08:50
• Resolved: 25/Nov/09 21:57

Description

Reducers fail while copying Map outputs with following exception

java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1539) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1432) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1285) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1216) ,Error:

Reducer's memory usage keeps on increasing and ultimately exceeds -Xmx value
I even tried with -Xmx6.5g to each reducer but it's still failing

While looking into the reducer logs, I found that reducers were doing shuffleInMemory every time, rather than doing shuffleOnDisk