Hive
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-7816

Enable map-join tests which Tez executes [Spark Branch]

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: Spark
    • Labels:
      None

      Description

        auto_join0.q,\
        auto_join1.q,\
        cross_join.q,\
        cross_product_check_1.q,\
        cross_product_check_2.q,\
      
      filter_join_breaktask.q,\
      filter_join_breaktask2.q
      
      1. HIVE-7816.2-spark.patch
        80 kB
        Chao Sun
      2. HIVE-7816.1-spark.patch
        60 kB
        Xuefu Zhang

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          113d 12h 13m 1 Rui Li 12/Dec/14 12:38
          Patch Available Patch Available Resolved Resolved
          2d 14h 29m 1 Xuefu Zhang 15/Dec/14 03:08
          Rui Li made changes -
          Link This issue relates to HIVE-9098 [ HIVE-9098 ]
          Hide
          Rui Li added a comment -

          That's OK. No worries, I'll take care of that.

          Show
          Rui Li added a comment - That's OK. No worries, I'll take care of that.
          Hide
          Xuefu Zhang added a comment -

          Sorry, Rui. Now we have to do it as a followup.

          Show
          Xuefu Zhang added a comment - Sorry, Rui. Now we have to do it as a followup.
          Hide
          Rui Li added a comment -

          Xuefu Zhang - shall we wait a little bit? That's just a couple of more lines of code. I'm running tests on my machine.

          Show
          Rui Li added a comment - Xuefu Zhang - shall we wait a little bit? That's just a couple of more lines of code. I'm running tests on my machine.
          Xuefu Zhang made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s spark-branch [ 12327352 ]
          Resolution Fixed [ 1 ]
          Hide
          Xuefu Zhang added a comment -

          Committed to Spark branch. Thanks, Rui.

          Show
          Xuefu Zhang added a comment - Committed to Spark branch. Thanks, Rui.
          Hide
          Xuefu Zhang added a comment -

          +1.

          Since the test has passed, I suggest we do that in a followup JIRA. I'm going to commit this patch shortly.

          Show
          Xuefu Zhang added a comment - +1. Since the test has passed, I suggest we do that in a followup JIRA. I'm going to commit this patch shortly.
          Hide
          Rui Li added a comment -

          Hi Xuefu Zhang yeah we have to deal with conditional task as well. Actually I missed that part when implementing SparkCrossProductCheck. Thanks for pointing out!

          Show
          Rui Li added a comment - Hi Xuefu Zhang yeah we have to deal with conditional task as well. Actually I missed that part when implementing SparkCrossProductCheck. Thanks for pointing out!
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12687107/HIVE-7816.2-spark.patch

          ERROR: -1 due to 3 failed/errored test(s), 7237 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/541/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/541/console
          Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-541/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12687107 - PreCommit-HIVE-SPARK-Build

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12687107/HIVE-7816.2-spark.patch ERROR: -1 due to 3 failed/errored test(s), 7237 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/541/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/541/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-541/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12687107 - PreCommit-HIVE-SPARK-Build
          Chao Sun made changes -
          Attachment HIVE-7816.2-spark.patch [ 12687107 ]
          Chao Sun made changes -
          Attachment HIVE-7816.2-spark.patch [ 12687106 ]
          Chao Sun made changes -
          Attachment HIVE-7816.2-spark.patch [ 12687106 ]
          Hide
          Chao Sun added a comment -

          Regenerated golden files.

          Show
          Chao Sun added a comment - Regenerated golden files.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12687094/HIVE-7816.1-spark.patch

          ERROR: -1 due to 22 failed/errored test(s), 7237 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join0
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join23
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_filters
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_nulls
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_12
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cross_join
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ctas
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join0
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join23
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_alt_syntax
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_1
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_3
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual1
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual3
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_lateral_view
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parallel_join0
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join5
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_multiinsert
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/540/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/540/console
          Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-540/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 22 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12687094 - PreCommit-HIVE-SPARK-Build

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12687094/HIVE-7816.1-spark.patch ERROR: -1 due to 22 failed/errored test(s), 7237 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join0 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join23 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_filters org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_nulls org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_12 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cross_join org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ctas org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join0 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join23 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_alt_syntax org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_lateral_view org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parallel_join0 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join5 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_multiinsert Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/540/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/540/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-540/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 22 tests failed This message is automatically generated. ATTACHMENT ID: 12687094 - PreCommit-HIVE-SPARK-Build
          Xuefu Zhang made changes -
          Attachment HIVE-7816.1-spark.patch [ 12686842 ]
          Xuefu Zhang made changes -
          Attachment HIVE-7816.1-spark.patch [ 12687094 ]
          Xuefu Zhang made changes -
          Attachment HIVE-7816.1-spark.patch [ 12686930 ]
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12686930/HIVE-7816.1-spark.patch

          ERROR: -1 due to 25 failed/errored test(s), 7265 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join0
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join23
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_filters
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_nulls
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_12
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cross_join
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join0
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join23
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_alt_syntax
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_1
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_3
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual1
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual3
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_filters
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_map_ppr
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_nulls
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parallel_join0
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join4
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join5
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_25
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_multiinsert
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/528/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/528/console
          Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-528/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 25 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12686930 - PreCommit-HIVE-SPARK-Build

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12686930/HIVE-7816.1-spark.patch ERROR: -1 due to 25 failed/errored test(s), 7265 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join0 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join23 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_filters org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_nulls org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_12 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cross_join org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join0 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join23 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_alt_syntax org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_filters org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_map_ppr org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_nulls org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parallel_join0 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join4 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join5 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_25 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_multiinsert Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/528/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/528/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-528/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 25 tests failed This message is automatically generated. ATTACHMENT ID: 12686930 - PreCommit-HIVE-SPARK-Build
          Hide
          Xuefu Zhang added a comment -

          Patch looks good. One question though, do we need to handle conditional task, since we added it for skew join?

          Show
          Xuefu Zhang added a comment - Patch looks good. One question though, do we need to handle conditional task, since we added it for skew join?
          Xuefu Zhang made changes -
          Attachment HIVE-7816.1-spark.patch [ 12686930 ]
          Rui Li made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Rui Li made changes -
          Attachment HIVE-7816.1-spark.patch [ 12686842 ]
          Hide
          Rui Li added a comment -

          This patch implements a spark-specific cross product checker SparkCrossProductCheck. The purpose of this resolver is to print warning messages for shuffle/map joins in case of cross product.
          The new golden files are not totally same as MR version, because we have different plans for the queries.

          Show
          Rui Li added a comment - This patch implements a spark-specific cross product checker SparkCrossProductCheck . The purpose of this resolver is to print warning messages for shuffle/map joins in case of cross product. The new golden files are not totally same as MR version, because we have different plans for the queries.
          Hide
          Rui Li added a comment -

          We already have golden files for filter_join_breaktask.q and filter_join_breaktask2.q, but they're not added to testconfiguration.properties.
          We actually haven't implemented cross product check for spark, so the test output is not correct. I'll look into this.

          Show
          Rui Li added a comment - We already have golden files for filter_join_breaktask.q and filter_join_breaktask2.q , but they're not added to testconfiguration.properties . We actually haven't implemented cross product check for spark, so the test output is not correct. I'll look into this.
          Hide
          Rui Li added a comment -

          OK I'm on it.

          Show
          Rui Li added a comment - OK I'm on it.
          Xuefu Zhang made changes -
          Assignee Rui Li [ lirui ]
          Hide
          Xuefu Zhang added a comment -

          It seems that not all test in the list are added for Spark yet. Rui Li, could you help take a look? Thanks.

          Show
          Xuefu Zhang added a comment - It seems that not all test in the list are added for Spark yet. Rui Li , could you help take a look? Thanks.
          Szehon Ho made changes -
          Summary Enable join tests which Tez executes Enable map-join tests which Tez executes [Spark Branch]
          Assignee Szehon Ho [ szehon ]
          Hide
          Szehon Ho added a comment -

          These tests are actually about map-join.

          I ran a few of them and they passed (the result match) , but found out its because they are actually running reduce-side join. Should probably revisit only once HIVE-7613 is done.

          Show
          Szehon Ho added a comment - These tests are actually about map-join. I ran a few of them and they passed (the result match) , but found out its because they are actually running reduce-side join. Should probably revisit only once HIVE-7613 is done.
          Szehon Ho made changes -
          Assignee Szehon Ho [ szehon ]
          Chinna Rao Lalam made changes -
          Description  
          {noformat}
            auto_join0.q,\
            auto_join1.q,\
            cross_join.q,\
            cross_product_check_1.q,\
            cross_product_check_2.q,\
          {noformat}
           
          {noformat}
            auto_join0.q,\
            auto_join1.q,\
            cross_join.q,\
            cross_product_check_1.q,\
            cross_product_check_2.q,\
          {noformat}

          {noformat}
          filter_join_breaktask.q,\
          filter_join_breaktask2.q
          {noformat}
          Brock Noland made changes -
          Field Original Value New Value
          Component/s Spark [ 12323200 ]
          Brock Noland created issue -

            People

            • Assignee:
              Rui Li
              Reporter:
              Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development