[HADOOP-3550] Reduce tasks failing with OOM

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.17.0
• Fix Version/s: 0.17.1
• Hadoop Flags: Reviewed
• Assignee: Christopher Douglas
• Reporter: Arun Murthy
• Created: 13/Jun/08 02:55
• Updated: 02/May/13 02:29
• Resolved: 18/Jun/08 22:45

Description

Map-Reduce jobs which worked with 0.16.* are failing with OOM with the following trace:

java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2786)
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at org.apache.hadoop.io.SequenceFile$UncompressedBytes.writeUncompressedBytes(SequenceFile.java:617)
        at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.readNextValue(ReduceTask.java:289)
        at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.next(ReduceTask.java:232)
        at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:311)
        at org.apache.hadoop.streaming.PipeReducer.reduce(PipeReducer.java:67)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:391)
        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)

Issue Links

• depends upon: HADOOP-3475 MapOutputBuffer allocates 4x as much space to record capacity as intended