[MAPREDUCE-21] NegativeArraySizeException in reducer with new api

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: task
• Assignee: Unassigned
• Reporter: Amareshwari Sriramadasu
• Created: 25/May/09 11:12
• Updated: 26/Feb/20 05:29
• Resolved: 27/Jun/16 05:05

Description

I observed one of the reducers failing with NegativeArraySizeException with new api.
The exception trace:

java.lang.NegativeArraySizeException
at org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:119)
at org.apache.hadoop.io.BytesWritable.setSize(BytesWritable.java:98)
at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:153)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
at org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:142)
at org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:121)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:189)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:542)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:409)
at org.apache.hadoop.mapred.Child.main(Child.java:159)

The corresponding line in ReduceContext is

line#142    key = keyDeserializer.deserialize(key);

Issue Links

• is duplicated by: HADOOP-11901 BytesWritable fails to support 2G chunks due to integer overflow
• is related to: MAPREDUCE-15 SequenceFile RecordReader should skip bad records