Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-5057

IndexOutOfBoundsException when pig reducer processOnePackageOutput

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.14.0, 0.15.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      pig-0.14/pig-0.15 cdh5.3.2

      Description

      When runing a pig job,the reducer throw out of bands exception,which leads the job failed.

      2016-11-14 15:31:04,752 WARN [main] org.apache.hadoop.io.compress.LzoCodec: org.apache.hadoop.io.compress.LzoCodec is deprecated. You should use com.hadoop.compression.lzo.LzoCodec instead to generate LZO compressed data.
      2016-11-14 15:32:08,300 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
      at java.util.ArrayList.rangeCheck(ArrayList.java:635)
      at java.util.ArrayList.get(ArrayList.java:411)
      at org.apache.pig.data.DefaultTuple.get(DefaultTuple.java:117)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.Packager.getValueTuple(Packager.java:234)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackage$PeekedBag$1.next(POPackage.java:435)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackage$PeekedBag$1.next(POPackage.java:408)
      at org.apache.pig.data.DefaultAbstractBag.addAll(DefaultAbstractBag.java:151)
      at org.apache.pig.data.DefaultAbstractBag.addAll(DefaultAbstractBag.java:137)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.Packager.attachInput(Packager.java:125)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackage.getNextTuple(POPackage.java:283)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:431)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:422)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:269)
      at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
      at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
      at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
      at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
      at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

        Attachments

        1. PIG-5057.patch
          0.9 kB
          shenxianqiang

          Activity

            People

            • Assignee:
              shenxianqiang shenxianqiang
              Reporter:
              shenxianqiang shenxianqiang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: