[MAPREDUCE-5031] Maps hitting IndexOutOfBoundsException for higher values of mapreduce.task.io.sort.mb

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.0.3-alpha, 0.23.5
• Component/s: mrv2
• Target Version/s: 0.23.7, 2.1.0-beta
• Assignee: Karthik Kambatla
• Reporter: Karthik Kambatla
• Created: 26/Feb/13 16:25
• Updated: 03/Nov/14 18:33
• Resolved: 04/Mar/13 23:48

Description

While trying to reproduce MAPREDUCE-5028 on trunk, ran into what seems to be a different issue. To reproduce:

Psuedo-dist mode: mapreduce.

{map,reduce}.memory.mb=2048, mapreduce.{map,reduce}

.java.opts=-Xmx2048m, mapreduce.task.io.sort.mb=1280

The map tasks fail with the following error:

Error: java.lang.IndexOutOfBoundsException at java.nio.Buffer.checkIndex(Buffer.java:512) at
 java.nio.ByteBufferAsIntBufferL.put(ByteBufferAsIntBufferL.java:113) at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1141) at 
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:686) at 
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at 
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at 
org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:47) at 
org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:36) at 
org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:757) at 
org.apache.hadoop.mapred.MapTask.run(MapTask.java:339) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158) at 
java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1488) at 
org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)

Issue Links

• relates to: MAPREDUCE-5032 MapTask.MapOutputBuffer contains arithmetic overflows
• relates to: MAPREDUCE-5028 Maps fail when io.sort.mb is set to high value