Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9106

improve the performance of null scan optimizer when several table scans share a physical path

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.15.0
    • Fix Version/s: 1.1.0
    • Component/s: Physical Optimizer
    • Labels:
      None

      Description

      Current fix HIVE-9053 addresses the correctness issue. The solution can be improved further when several table scans share a physical path.

      1. HIVE-9106.00.patch
        25 kB
        Pengcheng Xiong

        Issue Links

          Activity

          Hide
          pxiong Pengcheng Xiong added a comment -

          address the performance issue per Laljo John Pullokkaran's request

          Show
          pxiong Pengcheng Xiong added a comment - address the performance issue per Laljo John Pullokkaran 's request
          Hide
          pxiong Pengcheng Xiong added a comment -

          Ashutosh Chauhan, I discussed this with Laljo John Pullokkaran to improve the performance. As per Laljo John Pullokkaran's request, could you please kindly review it? Thanks!

          Show
          pxiong Pengcheng Xiong added a comment - Ashutosh Chauhan , I discussed this with Laljo John Pullokkaran to improve the performance. As per Laljo John Pullokkaran 's request, could you please kindly review it? Thanks!
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12687378/HIVE-9106.00.patch

          ERROR: -1 due to 4 failed/errored test(s), 6705 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby3_map_multi_distinct
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_partition_diff_num_cols
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_partition_diff_num_cols
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2092/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2092/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2092/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 4 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12687378 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12687378/HIVE-9106.00.patch ERROR: -1 due to 4 failed/errored test(s), 6705 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby3_map_multi_distinct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_partition_diff_num_cols org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_partition_diff_num_cols Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2092/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2092/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2092/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed This message is automatically generated. ATTACHMENT ID: 12687378 - PreCommit-HIVE-TRUNK-Build
          Hide
          pxiong Pengcheng Xiong added a comment -

          Ashutosh Chauhan, I studied the 4 failures. 2 out of the 4, vector_partition_diff_num_cols and vector_partition_diff_num_cols are probably due to Sergey's enable CBO patch (and the following Vikram's patch is not updated so that the column names are different). I run groupby3_map_multi_distinct.q and optimize_nullscan.q, both of them passed on my laptop. Thus, I think it is safe to check in. Laljo John Pullokkaran, please let us know if you have other opinions.

          Show
          pxiong Pengcheng Xiong added a comment - Ashutosh Chauhan , I studied the 4 failures. 2 out of the 4, vector_partition_diff_num_cols and vector_partition_diff_num_cols are probably due to Sergey's enable CBO patch (and the following Vikram's patch is not updated so that the column names are different). I run groupby3_map_multi_distinct.q and optimize_nullscan.q, both of them passed on my laptop. Thus, I think it is safe to check in. Laljo John Pullokkaran , please let us know if you have other opinions.
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          +1

          Show
          ashutoshc Ashutosh Chauhan added a comment - +1
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Pengcheng!

          Show
          ashutoshc Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Pengcheng!

            People

            • Assignee:
              pxiong Pengcheng Xiong
              Reporter:
              pxiong Pengcheng Xiong
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development