Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9141

HiveOnTez: mix of union all, distinct, group by generates error

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.15.0
    • Fix Version/s: 1.0.0
    • Component/s: Tez
    • Labels:
      None

      Description

      Here is the way to produce it:
      in Hive q test setting (with src table)
      set hive.execution.engine=tez;

      SELECT key, value FROM
      (
      SELECT key, value FROM src

      UNION ALL

      SELECT key, key as value FROM

      (
      SELECT distinct key FROM (

      SELECT key, value FROM
      (SELECT key, value FROM src
      UNION ALL
      SELECT key, value FROM src
      )t1
      group by key, value
      )t2
      )t3

      )t4
      group by key, value;

      will generate

      2014-12-16 23:19:13,593 ERROR ql.Driver (SessionState.java:printError(834)) - FAILED: ClassCastException org.apache.hadoop.hive.ql.plan.MapWork cannot be cast to org.apache.hadoop.hive.ql.plan.ReduceWork
      java.lang.ClassCastException: org.apache.hadoop.hive.ql.plan.MapWork cannot be cast to org.apache.hadoop.hive.ql.plan.ReduceWork
      at org.apache.hadoop.hive.ql.parse.GenTezWork.process(GenTezWork.java:361)
      at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
      at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
      at org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:87)
      at org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
      at org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
      at org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
      at org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
      at org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.startWalking(GenTezWorkWalker.java:69)
      at org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:368)
      at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:202)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10202)
      at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:224)
      at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:419)
      at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
      at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1107)
      at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1155)
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044)
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1034)
      at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:206)
      at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:158)
      at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:369)
      at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:304)
      at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:834)
      at org.apache.hadoop.hive.cli.TestMiniTezCliDriver.runTest(TestMiniTezCliDriver.java:136)
      at org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_uniontez2(TestMiniTezCliDriver.java:120)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        Issue Links

          Activity

          Hide
          pxiong Pengcheng Xiong added a comment -

          This will block HIVE-9039. Laljo John Pullokkaran and Vikram Dixit K, could you please take a look? The error remains in the most recent trunk (that is, after HIVE-9055 is checked in).

          Show
          pxiong Pengcheng Xiong added a comment - This will block HIVE-9039 . Laljo John Pullokkaran and Vikram Dixit K , could you please take a look? The error remains in the most recent trunk (that is, after HIVE-9055 is checked in).
          Hide
          pxiong Pengcheng Xiong added a comment -

          There is no error when it is run on MR.

          Show
          pxiong Pengcheng Xiong added a comment - There is no error when it is run on MR.
          Hide
          pxiong Pengcheng Xiong added a comment -

          And, the following queries got no error when they are run on Tez:

          SELECT key, value FROM(

          SELECT key, value FROM src

          UNION ALL

          SELECT key, value FROM src

          )t1 group by key, value;

          SELECT distinct key FROM (

          SELECT key, value FROM(

          SELECT key, value FROM src

          UNION ALL

          SELECT key, value FROM src

          )t1 group by key, value)t2;

          Show
          pxiong Pengcheng Xiong added a comment - And, the following queries got no error when they are run on Tez: SELECT key, value FROM( SELECT key, value FROM src UNION ALL SELECT key, value FROM src )t1 group by key, value; SELECT distinct key FROM ( SELECT key, value FROM( SELECT key, value FROM src UNION ALL SELECT key, value FROM src )t1 group by key, value)t2;
          Hide
          vikram.dixit Vikram Dixit K added a comment -

          Pengcheng Xiong I was able to run the query you mentioned above only getting a diff in the result. Can you try again with the latest changes and see it it works?

          Show
          vikram.dixit Vikram Dixit K added a comment - Pengcheng Xiong I was able to run the query you mentioned above only getting a diff in the result. Can you try again with the latest changes and see it it works?
          Hide
          navis Navis added a comment -

          Quick shot.

          Show
          navis Navis added a comment - Quick shot.
          Hide
          pxiong Pengcheng Xiong added a comment -

          Thanks Navis. I also tested the patch with other test cases and it works so far so good.

          Show
          pxiong Pengcheng Xiong added a comment - Thanks Navis . I also tested the patch with other test cases and it works so far so good.
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12687926/HIVE-9141.1.patch.txt

          ERROR: -1 due to 2 failed/errored test(s), 6714 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2121/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2121/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2121/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12687926 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12687926/HIVE-9141.1.patch.txt ERROR: -1 due to 2 failed/errored test(s), 6714 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2121/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2121/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2121/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12687926 - PreCommit-HIVE-TRUNK-Build
          Hide
          vikram.dixit Vikram Dixit K added a comment -

          Navis Nice patch. Simplifies the code as well. However, it looks like with this change, the followingWork can never be a union work because of moving the code. Do you see any way followingWork can be union work? I think we can remove that piece of the code as well as remove the getFollowingWorkIndex method as well. Thoughts?

          Show
          vikram.dixit Vikram Dixit K added a comment - Navis Nice patch. Simplifies the code as well. However, it looks like with this change, the followingWork can never be a union work because of moving the code. Do you see any way followingWork can be union work? I think we can remove that piece of the code as well as remove the getFollowingWorkIndex method as well. Thoughts?
          Hide
          ashutoshc Ashutosh Chauhan added a comment - - edited

          +1 This also fixes optimize_nullscan.q breakage introduced by HIVE-9055

          Show
          ashutoshc Ashutosh Chauhan added a comment - - edited +1 This also fixes optimize_nullscan.q breakage introduced by HIVE-9055
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Navis!
          Vikram Dixit K I have committed this since this one was also blocking HIVE-9106 in addition to resulting a breakage of test. If there is a way to improve on patch we can take that up on follow-on.

          Show
          ashutoshc Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Navis! Vikram Dixit K I have committed this since this one was also blocking HIVE-9106 in addition to resulting a breakage of test. If there is a way to improve on patch we can take that up on follow-on.
          Hide
          navis Navis added a comment -

          Vikram Dixit K My patch was just a quick shot (as I stated submitting the patch) thinking that parents of root should reference the work of root operator instead of currently evolving work. As it's committed already, you can try that in following issue. Frankly, I don't know well on tez compiling.

          Show
          navis Navis added a comment - Vikram Dixit K My patch was just a quick shot (as I stated submitting the patch) thinking that parents of root should reference the work of root operator instead of currently evolving work. As it's committed already, you can try that in following issue. Frankly, I don't know well on tez compiling.
          Hide
          thejas Thejas M Nair added a comment -

          Should we include this in hive 0.14.1 ?
          cc Vikram Dixit K

          Show
          thejas Thejas M Nair added a comment - Should we include this in hive 0.14.1 ? cc Vikram Dixit K
          Hide
          thejas Thejas M Nair added a comment -

          Updating release version for jiras resolved in 1.0.0 .

          Show
          thejas Thejas M Nair added a comment - Updating release version for jiras resolved in 1.0.0 .
          Hide
          thejas Thejas M Nair added a comment -

          This issue has been fixed in Apache Hive 1.0.0. If there is any issue with the fix, please open a new jira to address it.

          Show
          thejas Thejas M Nair added a comment - This issue has been fixed in Apache Hive 1.0.0. If there is any issue with the fix, please open a new jira to address it.

            People

            • Assignee:
              navis Navis
              Reporter:
              pxiong Pengcheng Xiong
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development