Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.0
    • Component/s: Tests, Tez
    • Labels:
      None

      Description

      Tez doesn't run all tests. But, if you run them, following tests fail with runt time exception pointing to bugs.

      • auto_join21.q
      • auto_join29.q
      • auto_join30.q
      • auto_join_filters.q
      • auto_join_nulls.q
      1. HIVE-9038.1.patch
        252 kB
        Vikram Dixit K
      2. HIVE-9038.2.patch
        252 kB
        Vikram Dixit K
      3. HIVE-9038.3.patch
        252 kB
        Vikram Dixit K

        Issue Links

          Activity

          Hide
          ashutoshc Ashutosh Chauhan added a comment -
          • auto_join21.q,auto_join29.q,auto_join_filters.q
            Vertex failed, vertexName=Map 3, vertexId=vertex_1417987826462_0002_2_02, diagnostics=[Task failed, taskId=task_1417987826462_0002_2_02_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"key":"238","value":"val_238"}
                    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
                    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
                    at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
                    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
                    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
                    at java.security.AccessController.doPrivileged(Native Method)
                    at javax.security.auth.Subject.doAs(Subject.java:415)
                    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
                    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
                    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                    at java.lang.Thread.run(Thread.java:745)
            Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"key":"238","value":"val_238"}
                    at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
                    at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
                    at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:294)
                    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163)
                    ... 13 more
            Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"key":"238","value":"val_238"}
                    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
                    at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
                    ... 16 more
            Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception: null
                    at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:314)
                    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
                    at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
                    at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
                    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
                    ... 17 more
            Caused by: java.lang.NullPointerException
                    at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getFilterTag(CommonJoinOperator.java:624)
                    at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.hasAnyFiltered(CommonJoinOperator.java:615)
                    at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:737)
                    at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:302)
                    ... 21 more
            
          • auto_join30.q
            TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"reducesinkkey0":"val_0"},"value":{"_col0":"0"}}
                    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
                    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
                    at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
                    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
                    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
                    at java.security.AccessController.doPrivileged(Native Method)
                    at javax.security.auth.Subject.doAs(Subject.java:415)
                    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
                    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
                    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                    at java.lang.Thread.run(Thread.java:745)
            Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"reducesinkkey0":"val_0"},"value":{"_col0":"0"}}
                    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:269)
                    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:168)
                    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163)
                    ... 13 more
            Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"reducesinkkey0":"val_0"},"value":{"_col0":"0"}}
                    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:337)
                    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:258)
                    ... 15 more
            Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception: java.lang.ArrayIndexOutOfBoundsException: 0
                    at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:314)
                    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
                    at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:85)
                    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:328)
                    ... 16 more
            Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: 0
                    at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:126)
                    at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:193)
                    at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:247)
                    ... 19 more
            Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
                    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct$SingleFieldGetter.init(LazyBinaryStruct.java:220)
                    at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$LazyBinaryKvWriter.updateStateByte(MapJoinBytesTableContainer.java:261)
                    at org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.put(BytesBytesMultiHashMap.java:230)
                    at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer.putRow(MapJoinBytesTableContainer.java:288)
                    at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:115)
            
          • auto_join_nulls.q
             TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: 1
                    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
                    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
                    at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
                    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
                    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
                    at java.security.AccessController.doPrivileged(Native Method)
                    at javax.security.auth.Subject.doAs(Subject.java:415)
                    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
                    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
                    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                    at java.lang.Thread.run(Thread.java:745)
            Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: 1
                    at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
                    at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
                    at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:294)
                    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163)
                    ... 13 more
            Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: 1
                    at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:126)
                    at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:193)
                    at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:219)
                    at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1051)
                    at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
                    at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
                    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:486)
                    at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
                    ... 16 more
            Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
                    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.byteArrayToShort(LazyBinaryUtils.java:101)
                    at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct$SingleFieldGetter.getShort(LazyBinaryStruct.java:242)
                    at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$LazyBinaryKvWriter.updateStateByte(MapJoinBytesTableContainer.java:262)
                    at org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.put(BytesBytesMultiHashMap.java:230)
                    at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer.putRow(MapJoinBytesTableContainer.java:288)
                    at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:115)
            
          Show
          ashutoshc Ashutosh Chauhan added a comment - auto_join21.q,auto_join29.q,auto_join_filters.q Vertex failed, vertexName=Map 3, vertexId=vertex_1417987826462_0002_2_02, diagnostics=[Task failed, taskId=task_1417987826462_0002_2_02_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row { "key" : "238" , "value" : "val_238" } at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row { "key" : "238" , "value" : "val_238" } at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:294) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163) ... 13 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row { "key" : "238" , "value" : "val_238" } at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83) ... 16 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception: null at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:314) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493) ... 17 more Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getFilterTag(CommonJoinOperator.java:624) at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.hasAnyFiltered(CommonJoinOperator.java:615) at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:737) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:302) ... 21 more auto_join30.q TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) { "key" :{ "reducesinkkey0" : "val_0" }, "value" :{ "_col0" : "0" }} at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) { "key" :{ "reducesinkkey0" : "val_0" }, "value" :{ "_col0" : "0" }} at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:269) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:168) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163) ... 13 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) { "key" :{ "reducesinkkey0" : "val_0" }, "value" :{ "_col0" : "0" }} at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:337) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:258) ... 15 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception: java.lang.ArrayIndexOutOfBoundsException: 0 at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:314) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:85) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:328) ... 16 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: 0 at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:126) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:193) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:247) ... 19 more Caused by: java.lang.ArrayIndexOutOfBoundsException: 0 at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct$SingleFieldGetter.init(LazyBinaryStruct.java:220) at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$LazyBinaryKvWriter.updateStateByte(MapJoinBytesTableContainer.java:261) at org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.put(BytesBytesMultiHashMap.java:230) at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer.putRow(MapJoinBytesTableContainer.java:288) at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:115) auto_join_nulls.q TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: 1 at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: 1 at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:294) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163) ... 13 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: 1 at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:126) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:193) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:219) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1051) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:486) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83) ... 16 more Caused by: java.lang.ArrayIndexOutOfBoundsException: 1 at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.byteArrayToShort(LazyBinaryUtils.java:101) at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct$SingleFieldGetter.getShort(LazyBinaryStruct.java:242) at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$LazyBinaryKvWriter.updateStateByte(MapJoinBytesTableContainer.java:262) at org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.put(BytesBytesMultiHashMap.java:230) at org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer.putRow(MapJoinBytesTableContainer.java:288) at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:115)
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          FYI : Vikram Dixit K Gunther Hagleitner
          Since there are three distinct stack traces, seems like there are 3 different bugs. Feel free to split this jira into three, as you see fit.

          Show
          ashutoshc Ashutosh Chauhan added a comment - FYI : Vikram Dixit K Gunther Hagleitner Since there are three distinct stack traces, seems like there are 3 different bugs. Feel free to split this jira into three, as you see fit.
          Hide
          navis Navis added a comment -

          Unlike MR, MapJoin in TEZ does not make filterTag in previous vertice but still want to read it. At first it sounded simple but I couldn't get all of those complex codes in MapJoinTableContainer, ReusableGetAdaptor, KeyValueHelper, KvSource,etc. Maybe Sergey Shelukhin can fix this at a glance.

          Show
          navis Navis added a comment - Unlike MR, MapJoin in TEZ does not make filterTag in previous vertice but still want to read it. At first it sounded simple but I couldn't get all of those complex codes in MapJoinTableContainer, ReusableGetAdaptor, KeyValueHelper, KvSource,etc. Maybe Sergey Shelukhin can fix this at a glance.
          Hide
          vikram.dixit Vikram Dixit K added a comment -

          This issue as Navis mentioned, stems from the fact that tez does not make filterTag. FilterTag is generated in case of MR by the HashTableSinkOperator which is not used in case of tez. The right solution would be to have a select operator that adds the filtertag to the value field so as to work without sacrificing the increased stages by moving this to a shuffle join instead of map join. However, since this only happens in the case where there are multiple joins on the same key with an outer join doing the filtering, I have this patch that changes the join to a shuffle join in this case for the time being so as to get away from the asserts. I will be raising a different jira for the good fix.

          Show
          vikram.dixit Vikram Dixit K added a comment - This issue as Navis mentioned, stems from the fact that tez does not make filterTag. FilterTag is generated in case of MR by the HashTableSinkOperator which is not used in case of tez. The right solution would be to have a select operator that adds the filtertag to the value field so as to work without sacrificing the increased stages by moving this to a shuffle join instead of map join. However, since this only happens in the case where there are multiple joins on the same key with an outer join doing the filtering, I have this patch that changes the join to a shuffle join in this case for the time being so as to get away from the asserts. I will be raising a different jira for the good fix.
          Hide
          sershe Sergey Shelukhin added a comment -

          I think default switch case should throw/assert... otherwise looks good. I didn't check the out files, I assume the query results are no different from non-tez

          Show
          sershe Sergey Shelukhin added a comment - I think default switch case should throw/assert... otherwise looks good. I didn't check the out files, I assume the query results are no different from non-tez
          Hide
          vikram.dixit Vikram Dixit K added a comment -

          Address Sergey's comments.

          Show
          vikram.dixit Vikram Dixit K added a comment - Address Sergey's comments.
          Hide
          sershe Sergey Shelukhin added a comment -

          +1 if tests pass

          Show
          sershe Sergey Shelukhin added a comment - +1 if tests pass
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12692388/HIVE-9038.3.patch

          ERROR: -1 due to 1 failed/errored test(s), 7315 tests executed
          Failed tests:

          org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2375/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2375/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2375/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12692388 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12692388/HIVE-9038.3.patch ERROR: -1 due to 1 failed/errored test(s), 7315 tests executed Failed tests: org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2375/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2375/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2375/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12692388 - PreCommit-HIVE-TRUNK-Build
          Hide
          vikram.dixit Vikram Dixit K added a comment -

          Test failure is unrelated. Same exception occurs without this change.

          Show
          vikram.dixit Vikram Dixit K added a comment - Test failure is unrelated. Same exception occurs without this change.
          Hide
          thejas Thejas M Nair added a comment -

          Updating release version for jiras resolved in 1.0.0 .

          Show
          thejas Thejas M Nair added a comment - Updating release version for jiras resolved in 1.0.0 .
          Hide
          thejas Thejas M Nair added a comment -

          This issue has been fixed in Apache Hive 1.0.0. If there is any issue with the fix, please open a new jira to address it.

          Show
          thejas Thejas M Nair added a comment - This issue has been fixed in Apache Hive 1.0.0. If there is any issue with the fix, please open a new jira to address it.

            People

            • Assignee:
              vikram.dixit Vikram Dixit K
              Reporter:
              ashutoshc Ashutosh Chauhan
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development