Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9832

Merge join followed by union and a map join in hive on tez fails.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.0.0, 1.2.0, 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: Tez
    • Labels:
      None

      Description

      select a.key, b.value from (select x.key as key, y.value as value from
      srcpart x join srcpart y on (x.key = y.key)
      union all
      select key, value from srcpart z) a join src b on (a.value = b.value);
      
      TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators: null
              at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
              at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
              at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
              at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
              at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
              at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
              at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
              at java.util.concurrent.FutureTask.run(FutureTask.java:262)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators: null
              at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:214)
              at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:177)
              ... 13 more
      Caused by: java.lang.NullPointerException
              at org.apache.hadoop.hive.ql.exec.MapJoinOperator.closeOp(MapJoinOperator.java:317)
              at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:598)
              at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
              at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
              at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
              at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
              at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:196)
              ... 14 more
      ]], Vertex failed as one or more tasks failed. failedTasks:1, Vertex vertex_1425055721029_0048_4_09 [Reducer 5] killed/failed due to:null]
      Vertex killed, vertexName=Reducer 7, vertexId=vertex_1425055721029_0048_4_11, diagnostics=[Vertex received Kill while in RUNNING state., Vertex killed as other vertex failed. failedTasks:0, Vertex vertex_1425055721029_0048_4_11 [Reducer 7] killed/failed due to:null]
      Vertex killed, vertexName=Reducer 4, vertexId=vertex_1425055721029_0048_4_07, diagnostics=[Vertex received Kill while in RUNNING state., Vertex killed as other vertex failed. failedTasks:0, Vertex vertex_1425055721029_0048_4_07 [Reducer 4] killed/failed due to:null]
      DAG failed due to vertex failure. failedVertices:1 killedVertices:2
      FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask
      
      1. HIVE-9832.1.patch
        589 kB
        Vikram Dixit K
      2. HIVE-9832.2.patch
        13 kB
        Vikram Dixit K
      3. HIVE-9832.3.patch
        14 kB
        Vikram Dixit K
      4. HIVE-9832.4.patch
        19 kB
        Vikram Dixit K

        Issue Links

          Activity

          Hide
          hagleitn Gunther Hagleitner added a comment -

          +1 although i think sum(hash(...)) would seriously reduce output size of the test.

          Show
          hagleitn Gunther Hagleitner added a comment - +1 although i think sum(hash(...)) would seriously reduce output size of the test.
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12702043/HIVE-9832.1.patch

          ERROR: -1 due to 1 failed/errored test(s), 7587 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2927/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2927/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2927/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12702043 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12702043/HIVE-9832.1.patch ERROR: -1 due to 1 failed/errored test(s), 7587 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2927/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2927/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2927/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12702043 - PreCommit-HIVE-TRUNK-Build
          Hide
          hagleitn Gunther Hagleitner added a comment -

          Vikram Dixit K can you take a look at the failure? it seems related.

          Show
          hagleitn Gunther Hagleitner added a comment - Vikram Dixit K can you take a look at the failure? it seems related.
          Hide
          vikram.dixit Vikram Dixit K added a comment -

          Looking at test failure.

          Show
          vikram.dixit Vikram Dixit K added a comment - Looking at test failure.
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12702220/HIVE-9832.2.patch

          ERROR: -1 due to 3 failed/errored test(s), 7586 tests executed
          Failed tests:

          TestCustomAuthentication - did not produce a TEST-*.xml file
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash
          org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2930/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2930/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2930/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12702220 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12702220/HIVE-9832.2.patch ERROR: -1 due to 3 failed/errored test(s), 7586 tests executed Failed tests: TestCustomAuthentication - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2930/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2930/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2930/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12702220 - PreCommit-HIVE-TRUNK-Build
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12702246/HIVE-9832.3.patch

          ERROR: -1 due to 2 failed/errored test(s), 7589 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2934/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2934/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2934/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12702246 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12702246/HIVE-9832.3.patch ERROR: -1 due to 2 failed/errored test(s), 7589 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2934/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2934/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2934/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12702246 - PreCommit-HIVE-TRUNK-Build
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12702391/HIVE-9832.4.patch

          ERROR: -1 due to 2 failed/errored test(s), 7574 tests executed
          Failed tests:

          TestSparkCliDriver-parallel_join1.q-ptf_general_queries.q-avro_joins.q-and-12-more - did not produce a TEST-*.xml file
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2941/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2941/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2941/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12702391 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12702391/HIVE-9832.4.patch ERROR: -1 due to 2 failed/errored test(s), 7574 tests executed Failed tests: TestSparkCliDriver-parallel_join1.q-ptf_general_queries.q-avro_joins.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2941/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2941/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2941/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12702391 - PreCommit-HIVE-TRUNK-Build
          Hide
          vikram.dixit Vikram Dixit K added a comment -

          Test Failures unrelated. They pass on my setup.

          Show
          vikram.dixit Vikram Dixit K added a comment - Test Failures unrelated. They pass on my setup.
          Hide
          vikram.dixit Vikram Dixit K added a comment -

          Brock Noland Ok for branch 1.1?

          Show
          vikram.dixit Vikram Dixit K added a comment - Brock Noland Ok for branch 1.1?
          Hide
          brocknoland Brock Noland added a comment -

          +1

          Show
          brocknoland Brock Noland added a comment - +1
          Hide
          sushanth Sushanth Sowmyan added a comment -

          This issue has been fixed and released as part of the 1.2.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          Show
          sushanth Sushanth Sowmyan added a comment - This issue has been fixed and released as part of the 1.2.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

            People

            • Assignee:
              vikram.dixit Vikram Dixit K
              Reporter:
              vikram.dixit Vikram Dixit K
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development