Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.0
    • Component/s: Tests, Tez
    • Labels:
      None

      Description

      Tez doesn't run all tests. But, if you run them, following tests fail with runt time exception pointing to bugs.

      • auto_join21.q
      • auto_join29.q
      • auto_join30.q
      • auto_join_filters.q
      • auto_join_nulls.q
      1. HIVE-9038.1.patch
        252 kB
        Vikram Dixit K
      2. HIVE-9038.2.patch
        252 kB
        Vikram Dixit K
      3. HIVE-9038.3.patch
        252 kB
        Vikram Dixit K

        Activity

        Hide
        ashutoshc Ashutosh Chauhan added a comment -
        • auto_join21.q,auto_join29.q,auto_join_filters.q
          Vertex failed, vertexName=Map 3, vertexId=vertex_1417987826462_0002_2_02, diagnostics=[Task failed, taskId=task_1417987826462_0002_2_02_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"key":"238","value":"val_238"}
                  at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
                  at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
                  at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
                  at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
                  at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at javax.security.auth.Subject.doAs(Subject.java:415)
                  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
                  at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
                  at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                  at java.lang.Thread.run(Thread.java:745)
          Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"key":"238","value":"val_238"}
                  at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
                  at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
                  at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:294)
                  at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163)
                  ... 13 more
          Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"key":"238","value":"val_238"}
                  at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
                  at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
                  ... 16 more
          Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception: null
                  at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:314)
                  at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
                  at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
                  at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
                  at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
                  ... 17 more
          Caused by: java.lang.NullPointerException
                  at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getFilterTag(CommonJoinOperator.java:624)
                  at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.hasAnyFiltered(CommonJoinOperator.java:615)
                  at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:737)
                  at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:302)
                  ... 21 more
          
        • auto_join30.q
          TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"reducesinkkey0":"val_0"},"value":{"_col0":"0"}}
                  at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
                  at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
                  at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
                  at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
                  at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at javax.security.auth.Subject.doAs(Subject.java:415)
                  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
                  at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
                  at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                  at java.lang.Thread.run(Thread.java:745)
          Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"reducesinkkey0":"val_0"},"value":{"_col0":"0"}}
                  at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:269)
                  at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:168)
                  at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163)
                  ... 13 more
          Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"reducesinkkey0":"val_0"},"value":{"_col0":"0"}}
                  at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:337)
                  at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:258)
                  ... 15 more
          Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception: java.lang.ArrayIndexOutOfBoundsException: 0
                  at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:314)
                  at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
                  at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:85)
                  at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:328)
                  ... 16 more
          Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: 0
                  at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:126)
                  at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:193)
                  at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:247)
                  ... 19 more
          Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
                  at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct$SingleFieldGetter.init(LazyBinaryStruct.java:220)
                  at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$LazyBinaryKvWriter.updateStateByte(MapJoinBytesTableContainer.java:261)
                  at org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.put(BytesBytesMultiHashMap.java:230)
                  at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer.putRow(MapJoinBytesTableContainer.java:288)
                  at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:115)
          
        • auto_join_nulls.q
           TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: 1
                  at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
                  at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
                  at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
                  at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
                  at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at javax.security.auth.Subject.doAs(Subject.java:415)
                  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
                  at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
                  at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                  at java.lang.Thread.run(Thread.java:745)
          Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: 1
                  at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
                  at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
                  at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:294)
                  at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163)
                  ... 13 more
          Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: 1
                  at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:126)
                  at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:193)
                  at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:219)
                  at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1051)
                  at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
                  at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
                  at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:486)
                  at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
                  ... 16 more
          Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
                  at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.byteArrayToShort(LazyBinaryUtils.java:101)
                  at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct$SingleFieldGetter.getShort(LazyBinaryStruct.java:242)
                  at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$LazyBinaryKvWriter.updateStateByte(MapJoinBytesTableContainer.java:262)
                  at org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.put(BytesBytesMultiHashMap.java:230)
                  at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer.putRow(MapJoinBytesTableContainer.java:288)
                  at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:115)
          
        Show
        ashutoshc Ashutosh Chauhan added a comment - auto_join21.q,auto_join29.q,auto_join_filters.q Vertex failed, vertexName=Map 3, vertexId=vertex_1417987826462_0002_2_02, diagnostics=[Task failed, taskId=task_1417987826462_0002_2_02_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row { "key" : "238" , "value" : "val_238" } at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row { "key" : "238" , "value" : "val_238" } at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:294) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163) ... 13 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row { "key" : "238" , "value" : "val_238" } at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83) ... 16 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception: null at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:314) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493) ... 17 more Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getFilterTag(CommonJoinOperator.java:624) at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.hasAnyFiltered(CommonJoinOperator.java:615) at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:737) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:302) ... 21 more auto_join30.q TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) { "key" :{ "reducesinkkey0" : "val_0" }, "value" :{ "_col0" : "0" }} at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) { "key" :{ "reducesinkkey0" : "val_0" }, "value" :{ "_col0" : "0" }} at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:269) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:168) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163) ... 13 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) { "key" :{ "reducesinkkey0" : "val_0" }, "value" :{ "_col0" : "0" }} at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:337) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:258) ... 15 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception: java.lang.ArrayIndexOutOfBoundsException: 0 at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:314) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:85) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:328) ... 16 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: 0 at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:126) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:193) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:247) ... 19 more Caused by: java.lang.ArrayIndexOutOfBoundsException: 0 at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct$SingleFieldGetter.init(LazyBinaryStruct.java:220) at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$LazyBinaryKvWriter.updateStateByte(MapJoinBytesTableContainer.java:261) at org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.put(BytesBytesMultiHashMap.java:230) at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer.putRow(MapJoinBytesTableContainer.java:288) at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:115) auto_join_nulls.q TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: 1 at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: 1 at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:294) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163) ... 13 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: 1 at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:126) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:193) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:219) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1051) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:486) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83) ... 16 more Caused by: java.lang.ArrayIndexOutOfBoundsException: 1 at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.byteArrayToShort(LazyBinaryUtils.java:101) at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct$SingleFieldGetter.getShort(LazyBinaryStruct.java:242) at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$LazyBinaryKvWriter.updateStateByte(MapJoinBytesTableContainer.java:262) at org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.put(BytesBytesMultiHashMap.java:230) at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer.putRow(MapJoinBytesTableContainer.java:288) at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:115)
        Hide
        ashutoshc Ashutosh Chauhan added a comment -

        FYI : Vikram Dixit K Gunther Hagleitner
        Since there are three distinct stack traces, seems like there are 3 different bugs. Feel free to split this jira into three, as you see fit.

        Show
        ashutoshc Ashutosh Chauhan added a comment - FYI : Vikram Dixit K Gunther Hagleitner Since there are three distinct stack traces, seems like there are 3 different bugs. Feel free to split this jira into three, as you see fit.
        Hide
        navis Navis added a comment -

        Unlike MR, MapJoin in TEZ does not make filterTag in previous vertice but still want to read it. At first it sounded simple but I couldn't get all of those complex codes in MapJoinTableContainer, ReusableGetAdaptor, KeyValueHelper, KvSource,etc. Maybe Sergey Shelukhin can fix this at a glance.

        Show
        navis Navis added a comment - Unlike MR, MapJoin in TEZ does not make filterTag in previous vertice but still want to read it. At first it sounded simple but I couldn't get all of those complex codes in MapJoinTableContainer, ReusableGetAdaptor, KeyValueHelper, KvSource,etc. Maybe Sergey Shelukhin can fix this at a glance.
        Hide
        vikram.dixit Vikram Dixit K added a comment -

        This issue as Navis mentioned, stems from the fact that tez does not make filterTag. FilterTag is generated in case of MR by the HashTableSinkOperator which is not used in case of tez. The right solution would be to have a select operator that adds the filtertag to the value field so as to work without sacrificing the increased stages by moving this to a shuffle join instead of map join. However, since this only happens in the case where there are multiple joins on the same key with an outer join doing the filtering, I have this patch that changes the join to a shuffle join in this case for the time being so as to get away from the asserts. I will be raising a different jira for the good fix.

        Show
        vikram.dixit Vikram Dixit K added a comment - This issue as Navis mentioned, stems from the fact that tez does not make filterTag. FilterTag is generated in case of MR by the HashTableSinkOperator which is not used in case of tez. The right solution would be to have a select operator that adds the filtertag to the value field so as to work without sacrificing the increased stages by moving this to a shuffle join instead of map join. However, since this only happens in the case where there are multiple joins on the same key with an outer join doing the filtering, I have this patch that changes the join to a shuffle join in this case for the time being so as to get away from the asserts. I will be raising a different jira for the good fix.
        Hide
        sershe Sergey Shelukhin added a comment -

        I think default switch case should throw/assert... otherwise looks good. I didn't check the out files, I assume the query results are no different from non-tez

        Show
        sershe Sergey Shelukhin added a comment - I think default switch case should throw/assert... otherwise looks good. I didn't check the out files, I assume the query results are no different from non-tez
        Hide
        vikram.dixit Vikram Dixit K added a comment -

        Address Sergey's comments.

        Show
        vikram.dixit Vikram Dixit K added a comment - Address Sergey's comments.
        Hide
        sershe Sergey Shelukhin added a comment -

        +1 if tests pass

        Show
        sershe Sergey Shelukhin added a comment - +1 if tests pass
        Hide
        hiveqa Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12692388/HIVE-9038.3.patch

        ERROR: -1 due to 1 failed/errored test(s), 7315 tests executed
        Failed tests:

        org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2375/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2375/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2375/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 1 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12692388 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12692388/HIVE-9038.3.patch ERROR: -1 due to 1 failed/errored test(s), 7315 tests executed Failed tests: org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2375/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2375/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2375/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12692388 - PreCommit-HIVE-TRUNK-Build
        Hide
        vikram.dixit Vikram Dixit K added a comment -

        Test failure is unrelated. Same exception occurs without this change.

        Show
        vikram.dixit Vikram Dixit K added a comment - Test failure is unrelated. Same exception occurs without this change.
        Hide
        thejas Thejas M Nair added a comment -

        Updating release version for jiras resolved in 1.0.0 .

        Show
        thejas Thejas M Nair added a comment - Updating release version for jiras resolved in 1.0.0 .
        Hide
        thejas Thejas M Nair added a comment -

        This issue has been fixed in Apache Hive 1.0.0. If there is any issue with the fix, please open a new jira to address it.

        Show
        thejas Thejas M Nair added a comment - This issue has been fixed in Apache Hive 1.0.0. If there is any issue with the fix, please open a new jira to address it.

          People

          • Assignee:
            vikram.dixit Vikram Dixit K
            Reporter:
            ashutoshc Ashutosh Chauhan
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development