Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-7816

Enable map-join tests which Tez executes [Spark Branch]

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: Spark
    • Labels:
      None

      Description

        auto_join0.q,\
        auto_join1.q,\
        cross_join.q,\
        cross_product_check_1.q,\
        cross_product_check_2.q,\
      
      filter_join_breaktask.q,\
      filter_join_breaktask2.q
      
      1. HIVE-7816.2-spark.patch
        80 kB
        Chao Sun
      2. HIVE-7816.1-spark.patch
        60 kB
        Xuefu Zhang

        Issue Links

          Activity

          Hide
          lirui Rui Li added a comment -

          That's OK. No worries, I'll take care of that.

          Show
          lirui Rui Li added a comment - That's OK. No worries, I'll take care of that.
          Hide
          xuefuz Xuefu Zhang added a comment -

          Sorry, Rui. Now we have to do it as a followup.

          Show
          xuefuz Xuefu Zhang added a comment - Sorry, Rui. Now we have to do it as a followup.
          Hide
          lirui Rui Li added a comment -

          Xuefu Zhang - shall we wait a little bit? That's just a couple of more lines of code. I'm running tests on my machine.

          Show
          lirui Rui Li added a comment - Xuefu Zhang - shall we wait a little bit? That's just a couple of more lines of code. I'm running tests on my machine.
          Hide
          xuefuz Xuefu Zhang added a comment -

          Committed to Spark branch. Thanks, Rui.

          Show
          xuefuz Xuefu Zhang added a comment - Committed to Spark branch. Thanks, Rui.
          Hide
          xuefuz Xuefu Zhang added a comment -

          +1.

          Since the test has passed, I suggest we do that in a followup JIRA. I'm going to commit this patch shortly.

          Show
          xuefuz Xuefu Zhang added a comment - +1. Since the test has passed, I suggest we do that in a followup JIRA. I'm going to commit this patch shortly.
          Hide
          lirui Rui Li added a comment -

          Hi Xuefu Zhang yeah we have to deal with conditional task as well. Actually I missed that part when implementing SparkCrossProductCheck. Thanks for pointing out!

          Show
          lirui Rui Li added a comment - Hi Xuefu Zhang yeah we have to deal with conditional task as well. Actually I missed that part when implementing SparkCrossProductCheck. Thanks for pointing out!
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12687107/HIVE-7816.2-spark.patch

          ERROR: -1 due to 3 failed/errored test(s), 7237 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/541/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/541/console
          Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-541/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12687107 - PreCommit-HIVE-SPARK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12687107/HIVE-7816.2-spark.patch ERROR: -1 due to 3 failed/errored test(s), 7237 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/541/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/541/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-541/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12687107 - PreCommit-HIVE-SPARK-Build
          Hide
          csun Chao Sun added a comment -

          Regenerated golden files.

          Show
          csun Chao Sun added a comment - Regenerated golden files.
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12687094/HIVE-7816.1-spark.patch

          ERROR: -1 due to 22 failed/errored test(s), 7237 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join0
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join23
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_filters
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_nulls
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_12
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cross_join
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ctas
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join0
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join23
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_alt_syntax
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_1
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_3
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual1
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual3
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_lateral_view
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parallel_join0
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join5
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_multiinsert
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/540/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/540/console
          Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-540/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 22 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12687094 - PreCommit-HIVE-SPARK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12687094/HIVE-7816.1-spark.patch ERROR: -1 due to 22 failed/errored test(s), 7237 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join0 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join23 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_filters org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_nulls org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_12 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cross_join org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ctas org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join0 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join23 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_alt_syntax org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_lateral_view org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parallel_join0 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join5 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_multiinsert Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/540/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/540/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-540/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 22 tests failed This message is automatically generated. ATTACHMENT ID: 12687094 - PreCommit-HIVE-SPARK-Build
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12686930/HIVE-7816.1-spark.patch

          ERROR: -1 due to 25 failed/errored test(s), 7265 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join0
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join23
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_filters
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_nulls
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_12
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cross_join
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join0
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join23
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_alt_syntax
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_1
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_3
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual1
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual3
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_filters
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_map_ppr
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_nulls
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parallel_join0
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join4
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join5
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_25
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_multiinsert
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/528/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/528/console
          Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-528/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 25 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12686930 - PreCommit-HIVE-SPARK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12686930/HIVE-7816.1-spark.patch ERROR: -1 due to 25 failed/errored test(s), 7265 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join0 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join23 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_filters org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_nulls org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_12 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cross_join org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join0 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join23 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_alt_syntax org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_filters org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_map_ppr org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_nulls org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parallel_join0 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join4 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join5 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_25 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_multiinsert Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/528/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/528/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-528/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 25 tests failed This message is automatically generated. ATTACHMENT ID: 12686930 - PreCommit-HIVE-SPARK-Build
          Hide
          xuefuz Xuefu Zhang added a comment -

          Patch looks good. One question though, do we need to handle conditional task, since we added it for skew join?

          Show
          xuefuz Xuefu Zhang added a comment - Patch looks good. One question though, do we need to handle conditional task, since we added it for skew join?
          Hide
          lirui Rui Li added a comment -

          This patch implements a spark-specific cross product checker SparkCrossProductCheck. The purpose of this resolver is to print warning messages for shuffle/map joins in case of cross product.
          The new golden files are not totally same as MR version, because we have different plans for the queries.

          Show
          lirui Rui Li added a comment - This patch implements a spark-specific cross product checker SparkCrossProductCheck . The purpose of this resolver is to print warning messages for shuffle/map joins in case of cross product. The new golden files are not totally same as MR version, because we have different plans for the queries.
          Hide
          lirui Rui Li added a comment -

          We already have golden files for filter_join_breaktask.q and filter_join_breaktask2.q, but they're not added to testconfiguration.properties.
          We actually haven't implemented cross product check for spark, so the test output is not correct. I'll look into this.

          Show
          lirui Rui Li added a comment - We already have golden files for filter_join_breaktask.q and filter_join_breaktask2.q , but they're not added to testconfiguration.properties . We actually haven't implemented cross product check for spark, so the test output is not correct. I'll look into this.
          Hide
          lirui Rui Li added a comment -

          OK I'm on it.

          Show
          lirui Rui Li added a comment - OK I'm on it.
          Hide
          xuefuz Xuefu Zhang added a comment -

          It seems that not all test in the list are added for Spark yet. Rui Li, could you help take a look? Thanks.

          Show
          xuefuz Xuefu Zhang added a comment - It seems that not all test in the list are added for Spark yet. Rui Li , could you help take a look? Thanks.
          Hide
          szehon Szehon Ho added a comment -

          These tests are actually about map-join.

          I ran a few of them and they passed (the result match) , but found out its because they are actually running reduce-side join. Should probably revisit only once HIVE-7613 is done.

          Show
          szehon Szehon Ho added a comment - These tests are actually about map-join. I ran a few of them and they passed (the result match) , but found out its because they are actually running reduce-side join. Should probably revisit only once HIVE-7613 is done.

            People

            • Assignee:
              lirui Rui Li
              Reporter:
              brocknoland Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development