Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5506

Hadoop-1.1.1 occurs ArrayIndexOutOfBoundsException with MultithreadedMapRunner

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Won't Fix
    • Affects Version/s: 1.1.1
    • Fix Version/s: None
    • Component/s: mrv1
    • Labels:
      None
    • Environment:

      RHEL 6.3 x86_64

      Description

      After I set:

      • 'jobConf.setMapRunnerClass(MultithreadedMapRunner.class);' in MR app
      • 'mapred.map.multithreadedrunner.threads = 2' in mapred-site.xml

      A simple MR app failed as its Map task encountered ArrayIndexOutOfBoundsException as below(please ignore the line numbers in the exception as I added some log print codes):
      java.lang.ArrayIndexOutOfBoundsException
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1331)
      at java.io.DataOutputStream.write(DataOutputStream.java:101)
      at org.apache.hadoop.io.Text.write(Text.java:282)
      at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
      at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1060)
      at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:591)
      at study.hadoop.mapreduce.sample.WordCount$Map.map(WordCount.java:41)
      at study.hadoop.mapreduce.sample.WordCount$Map.map(WordCount.java:1)
      at org.apache.hadoop.mapred.lib.MultithreadedMapRunner$MapperInvokeRunable.run(MultithreadedMapRunner.java:231)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
      at java.lang.Thread.run(Thread.java:738)

      And the exception happens on line 'System.arraycopy(b, off, kvbuffer, bufindex, len)' in MapTask.java#MapOutputBuffer#Buffer#write(). When the exception occurs, 'b.length=4' but 'len=9'.

      Btw, if I set 'mapred.map.multithreadedrunner.threads = 1', no exception happened. So it should be an issue caused by multiple threads.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              sam liu sam liu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: