Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-8888

Mapjoin with LateralViewJoin generates wrong plan in Tez

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0, 0.14.0, 0.13.1, 0.15.0
    • Fix Version/s: 1.0.0
    • Component/s: None
    • Labels:
      None

      Description

      Queries like these

      with sub1 as
      (select aid, avalue from expod1 lateral view explode(av) avs as avalue ),
      sub2 as
      (select bid, bvalue from expod2 lateral view explode(bv) bvs as bvalue)
      select sub1.aid, sub1.avalue, sub2.bvalue
      from sub1,sub2
      where sub1.aid=sub2.bid;
      

      generates twice the number of rows in Tez when compared to MR.

      1. HIVE-8888.1.patch
        20 kB
        Prasanth Jayachandran
      2. HIVE-8888.2.patch
        20 kB
        Prasanth Jayachandran
      3. HIVE-8888.3.patch
        19 kB
        Prasanth Jayachandran
      4. HIVE-8888.4.patch
        20 kB
        Gunther Hagleitner
      5. HIVE-8888.5.patch
        9 kB
        Gunther Hagleitner
      6. HIVE-8888-branch.5.patch
        1 kB
        Prasanth Jayachandran

        Issue Links

          Activity

          Hide
          prasanth_j Prasanth Jayachandran added a comment -
          Show
          prasanth_j Prasanth Jayachandran added a comment - Gunther Hagleitner Can you take a look at the fix? https://reviews.apache.org/r/28086/
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12681705/HIVE-8888.1.patch

          ERROR: -1 due to 5 failed/errored test(s), 6689 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_10
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_smb_main
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1810/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1810/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1810/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 5 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12681705 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12681705/HIVE-8888.1.patch ERROR: -1 due to 5 failed/errored test(s), 6689 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_10 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_smb_main org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1810/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1810/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1810/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed This message is automatically generated. ATTACHMENT ID: 12681705 - PreCommit-HIVE-TRUNK-Build
          Hide
          prasanth_j Prasanth Jayachandran added a comment -

          Wrong if condition in previous patch.

          Show
          prasanth_j Prasanth Jayachandran added a comment - Wrong if condition in previous patch.
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12681810/HIVE-8888.2.patch

          ERROR: -1 due to 1 failed/errored test(s), 6648 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1815/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1815/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1815/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12681810 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12681810/HIVE-8888.2.patch ERROR: -1 due to 1 failed/errored test(s), 6648 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1815/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1815/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1815/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12681810 - PreCommit-HIVE-TRUNK-Build
          Hide
          prasanth_j Prasanth Jayachandran added a comment -

          This patch bails out when the operator tree is visited again from the same root.

          Show
          prasanth_j Prasanth Jayachandran added a comment - This patch bails out when the operator tree is visited again from the same root.
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12682032/HIVE-8888.3.patch

          ERROR: -1 due to 3 failed/errored test(s), 6648 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_join_nullsafe
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_ptf
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1828/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1828/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1828/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12682032 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12682032/HIVE-8888.3.patch ERROR: -1 due to 3 failed/errored test(s), 6648 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_join_nullsafe org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_ptf Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1828/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1828/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1828/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12682032 - PreCommit-HIVE-TRUNK-Build
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12682247/HIVE-8888.4.patch

          ERROR: -1 due to 2 failed/errored test(s), 6649 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_covar_pop
          org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1837/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1837/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1837/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12682247 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12682247/HIVE-8888.4.patch ERROR: -1 due to 2 failed/errored test(s), 6649 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_covar_pop org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1837/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1837/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1837/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12682247 - PreCommit-HIVE-TRUNK-Build
          Hide
          hagleitn Gunther Hagleitner added a comment -

          Don't think the test failures are related.

          Prasanth Jayachandran thoughts? I'm +1 on the last patch.

          Show
          hagleitn Gunther Hagleitner added a comment - Don't think the test failures are related. Prasanth Jayachandran thoughts? I'm +1 on the last patch.
          Hide
          prasanth_j Prasanth Jayachandran added a comment -

          Gunther Hagleitner Even I don't think the test failure is related. The code changes should not affect TestCliDriver tests. I ran the test locally and it ran successfully.

          Also can we have this for 0.14.1?

          Show
          prasanth_j Prasanth Jayachandran added a comment - Gunther Hagleitner Even I don't think the test failure is related. The code changes should not affect TestCliDriver tests. I ran the test locally and it ran successfully. Also can we have this for 0.14.1?
          Hide
          prasanth_j Prasanth Jayachandran added a comment -

          Gunther Hagleitner/Ashutosh Chauhan Should this go into 0.14.1?

          Show
          prasanth_j Prasanth Jayachandran added a comment - Gunther Hagleitner / Ashutosh Chauhan Should this go into 0.14.1?
          Hide
          prasanth_j Prasanth Jayachandran added a comment -

          Committed to trunk

          Show
          prasanth_j Prasanth Jayachandran added a comment - Committed to trunk
          Hide
          hagleitn Gunther Hagleitner added a comment -

          This is causing incorrect query results. Should go into 0.14.1.

          Show
          hagleitn Gunther Hagleitner added a comment - This is causing incorrect query results. Should go into 0.14.1.
          Hide
          hagleitn Gunther Hagleitner added a comment -

          Committed to branch.

          Show
          hagleitn Gunther Hagleitner added a comment - Committed to branch.
          Hide
          hagleitn Gunther Hagleitner added a comment -

          The original patch doesn't actually work. The problem is that we still collect the mj during visiting the nodes. We need to clear that before bailing out of work generation.

          Show
          hagleitn Gunther Hagleitner added a comment - The original patch doesn't actually work. The problem is that we still collect the mj during visiting the nodes. We need to clear that before bailing out of work generation.
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12684137/HIVE-8888.5.patch

          ERROR: -1 due to 3 failed/errored test(s), 6694 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_aggregate
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mapjoin_mapjoin
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1930/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1930/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1930/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12684137 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12684137/HIVE-8888.5.patch ERROR: -1 due to 3 failed/errored test(s), 6694 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_aggregate org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mapjoin_mapjoin org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1930/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1930/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1930/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12684137 - PreCommit-HIVE-TRUNK-Build
          Hide
          hagleitn Gunther Hagleitner added a comment -

          failures are unrelated.

          Show
          hagleitn Gunther Hagleitner added a comment - failures are unrelated.
          Hide
          brocknoland Brock Noland added a comment -

          Gunther Hagleitner due to HIVE-8964 the latest ptest run won't actually test lvj_mapjoin.q if you've verified it works locally, I will enable that test after the latest patch is committed.

          Show
          brocknoland Brock Noland added a comment - Gunther Hagleitner due to HIVE-8964 the latest ptest run won't actually test lvj_mapjoin.q if you've verified it works locally, I will enable that test after the latest patch is committed.
          Hide
          hagleitn Gunther Hagleitner added a comment -

          Brock Noland I have verified that lvj_mapjoin works locally. Thanks.

          Show
          hagleitn Gunther Hagleitner added a comment - Brock Noland I have verified that lvj_mapjoin works locally. Thanks.
          Hide
          prasanth_j Prasanth Jayachandran added a comment -

          Last patch looks good to me. +1

          Show
          prasanth_j Prasanth Jayachandran added a comment - Last patch looks good to me. +1
          Hide
          prasanth_j Prasanth Jayachandran added a comment -

          Committed this patch to trunk. lvj_mapjoin.q ran successfully locally for me too. Brock Noland Can you reenable lvj_mapjoin.q test to see if it runs successfully now?

          Show
          prasanth_j Prasanth Jayachandran added a comment - Committed this patch to trunk. lvj_mapjoin.q ran successfully locally for me too. Brock Noland Can you reenable lvj_mapjoin.q test to see if it runs successfully now?
          Hide
          prasanth_j Prasanth Jayachandran added a comment -

          HIVE-8707 fixed ordering issues in hashmap between JDK8 vs older version. The test diff changes in .5 patch is not required for the branch. Attaching patch for the branch 0.14.1 which is same as Gunther's .5 patch but without test diff changes.

          Show
          prasanth_j Prasanth Jayachandran added a comment - HIVE-8707 fixed ordering issues in hashmap between JDK8 vs older version. The test diff changes in .5 patch is not required for the branch. Attaching patch for the branch 0.14.1 which is same as Gunther's .5 patch but without test diff changes.
          Hide
          prasanth_j Prasanth Jayachandran added a comment -

          Committed patch with Gunther Hagleitner's fix to branch-0.14

          Show
          prasanth_j Prasanth Jayachandran added a comment - Committed patch with Gunther Hagleitner 's fix to branch-0.14
          Hide
          prasanth_j Prasanth Jayachandran added a comment -

          Brock Noland I have committed this patch to trunk and branch-0.14.1. Please reopen this issue if lvj_mapjoin.q hangs again in test run.

          Show
          prasanth_j Prasanth Jayachandran added a comment - Brock Noland I have committed this patch to trunk and branch-0.14.1. Please reopen this issue if lvj_mapjoin.q hangs again in test run.
          Hide
          brocknoland Brock Noland added a comment -

          I have just enabled the test. I will keep an eye on the tests.

          Show
          brocknoland Brock Noland added a comment - I have just enabled the test. I will keep an eye on the tests.
          Hide
          thejas Thejas M Nair added a comment -

          Updating release version for jiras resolved in 1.0.0 .

          Show
          thejas Thejas M Nair added a comment - Updating release version for jiras resolved in 1.0.0 .
          Hide
          thejas Thejas M Nair added a comment -

          This issue has been fixed in Apache Hive 1.0.0. If there is any issue with the fix, please open a new jira to address it.

          Show
          thejas Thejas M Nair added a comment - This issue has been fixed in Apache Hive 1.0.0. If there is any issue with the fix, please open a new jira to address it.

            People

            • Assignee:
              prasanth_j Prasanth Jayachandran
              Reporter:
              prasanth_j Prasanth Jayachandran
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development