[HIVE-10047] LLAP: VectorMapJoinOperator gets an over-flow on batchSize of 1024

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: llap
• Fix Version/s: llap
• Component/s: llap
• Assignee: Matt McCline
• Reporter: Gopal Vijayaraghavan
• Created: 21/Mar/15 02:03
• Updated: 17/Feb/16 01:32
• Resolved: 26/Nov/15 02:29

Description

Simple LLAP queries on constrained resources runs into an exception which suggests that the

Caused by: java.lang.ArrayIndexOutOfBoundsException: 1024
        at org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory$VectorLongColumnAssign.assignLong(VectorColumnAssignFactory.java:113)
        at org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory$9.assignObjectValue(VectorColumnAssignFactory.java:293)
        at org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:196)
        at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:653)
        at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:656)
        at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:752)
        at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:316)
        ... 22 more

The relevant line is due to the check for a full output-batch being outside of the loop here - looks like it can be triggered during MxN joins where there are more values than there were input rows in the input batch.

    for (int i=0; i<values.length; ++i) {
      vcas[i].assignObjectValue(values[i], outputBatch.size);
    }
   ++outputBatch.size;
    if (outputBatch.size == VectorizedRowBatch.DEFAULT_SIZE) {
      flushOutput();
    }

Issue Links

• relates to: HIVE-10335 LLAP: IndexOutOfBound in MapJoinOperator