Hive
  1. Hive
  2. HIVE-6369

ORC Writer (int RLE v2) fails with ArrayIndexOutOfBounds

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.13.0
    • Component/s: File Formats
    • Labels:
    • Environment:

      hadoop-2.2 + hive-0.13-trunk (f1807ede)

      Description

      The ORC writer for store_sales TPC-DS table fails with

      2014-01-30 09:23:07,819 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.ArrayIndexOutOfBoundsException: 2
      	at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerWriterV2.preparePatchedBlob(RunLengthIntegerWriterV2.java:593)
      	at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerWriterV2.determineEncoding(RunLengthIntegerWriterV2.java:541)
      	at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerWriterV2.flush(RunLengthIntegerWriterV2.java:682)
      	at org.apache.hadoop.hive.ql.io.orc.WriterImpl$IntegerTreeWriter.writeStripe(WriterImpl.java:752)
      	at org.apache.hadoop.hive.ql.io.orc.WriterImpl$StructTreeWriter.writeStripe(WriterImpl.java:1330)
      	at org.apache.hadoop.hive.ql.io.orc.WriterImpl.flushStripe(WriterImpl.java:1699)
      	at org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:1868)
      

        Issue Links

          Activity

          Hide
          Prasanth J added a comment -

          Duplicate of HIVE-6382

          Show
          Prasanth J added a comment - Duplicate of HIVE-6382
          Hide
          Prasanth J added a comment -

          Assigning this bug to me.

          Show
          Prasanth J added a comment - Assigning this bug to me.
          Hide
          Prasanth J added a comment -

          Awesome! Will look into it.

          Show
          Prasanth J added a comment - Awesome! Will look into it.
          Hide
          Gopal V added a comment -

          Yes, this is from a recent build.

          But to make absolutely sure I disassembled the RLE writer from the built jar

          private void preparePatchedBlob();
            Code:
             0:	lconst_1
             1:	aload_0
             2:	getfield	#142; //Field brBits95p:I
             5:	lshl
             6:	lconst_1
             7:	lsub
             8:	lstore_1
          

          I can see that the operations are all lshl (long shift left) instead of ishl (integer shift left).

          Show
          Gopal V added a comment - Yes, this is from a recent build. But to make absolutely sure I disassembled the RLE writer from the built jar private void preparePatchedBlob(); Code: 0: lconst_1 1: aload_0 2: getfield #142; //Field brBits95p:I 5: lshl 6: lconst_1 7: lsub 8: lstore_1 I can see that the operations are all lshl (long shift left) instead of ishl (integer shift left).
          Hide
          Prasanth J added a comment -

          If it happens with the trunk, then my suspicion is that there are 64 bit values which causes overflow in mask creation. I will take a look at it once HIVE-5991 is confirmed.

          Show
          Prasanth J added a comment - If it happens with the trunk, then my suspicion is that there are 64 bit values which causes overflow in mask creation. I will take a look at it once HIVE-5991 is confirmed.
          Hide
          Prasanth J added a comment -

          Gopal V Can you please check if HIVE-5991 patch is available in your test environment?

          Show
          Prasanth J added a comment - Gopal V Can you please check if HIVE-5991 patch is available in your test environment?
          Hide
          Gopal V added a comment -
          2014-01-30 10:56:50,753 INFO  [main] mapreduce.Job (Job.java:printTaskEvents(1424)) - Task Id : attempt_1390984584295_0015_r_004034_0, Status : FAILED
          Error: java.lang.ArrayIndexOutOfBoundsException: 26
                  at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerWriterV2.preparePatchedBlob(RunLengthIntegerWriterV2.java:593)
                  at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerWriterV2.determineEncoding(RunLengthIntegerWriterV2.java:541)
                  at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerWriterV2.flush(RunLengthIntegerWriterV2.java:682)
                  at org.apache.hadoop.hive.ql.io.orc.WriterImpl$IntegerTreeWriter.writeStripe(WriterImpl.java:752)
                  at org.apache.hadoop.hive.ql.io.orc.WriterImpl$StructTreeWriter.writeStripe(WriterImpl.java:1330)
          
          Show
          Gopal V added a comment - 2014-01-30 10:56:50,753 INFO [main] mapreduce.Job (Job.java:printTaskEvents(1424)) - Task Id : attempt_1390984584295_0015_r_004034_0, Status : FAILED Error: java.lang.ArrayIndexOutOfBoundsException: 26 at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerWriterV2.preparePatchedBlob(RunLengthIntegerWriterV2.java:593) at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerWriterV2.determineEncoding(RunLengthIntegerWriterV2.java:541) at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerWriterV2.flush(RunLengthIntegerWriterV2.java:682) at org.apache.hadoop.hive.ql.io.orc.WriterImpl$IntegerTreeWriter.writeStripe(WriterImpl.java:752) at org.apache.hadoop.hive.ql.io.orc.WriterImpl$StructTreeWriter.writeStripe(WriterImpl.java:1330)

            People

            • Assignee:
              Prasanth J
              Reporter:
              Gopal V
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development