Details

    • Improvement
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • None
    • 2.0.0
    • Hive
    • None

    Description

      Currently, VectorReduceSinkOperator is a pass-thru to ReduceSinkOperator, so we incur object inspector costs.

      Native vectorization will not use object inspectors and allocate memory up front that will be reused for each batch.

      Attachments

        1. HIVE-12290.01.patch
          499 kB
          Matt McCline
        2. HIVE-12290.02.patch
          501 kB
          Matt McCline
        3. HIVE-12290.03.patch
          522 kB
          Matt McCline
        4. HIVE-12290.04.patch
          522 kB
          Matt McCline
        5. HIVE-12290.05.patch
          524 kB
          Matt McCline
        6. HIVE-12290.06.patch
          524 kB
          Matt McCline

        Issue Links

          Activity

            sladymon Shannon Ladymon added a comment - Doc done: Configuration Properties - hive.vectorized.execution.reducesink.new.enabled
            leftyl Lefty Leverenz added a comment -

            Matt reverted the addition of HiveConf.java.orig with commit 9ba2cdfd9a8eae371048de3fa92131517d8f8d6d.

            leftyl Lefty Leverenz added a comment - Matt reverted the addition of HiveConf.java.orig with commit 9ba2cdfd9a8eae371048de3fa92131517d8f8d6d.
            leftyl Lefty Leverenz added a comment -

            Doc note: This adds hive.vectorized.execution.reducesink.new.enabled to HiveConf.java, so it needs to be documented in the wiki for release 2.0.0.

            Adding a TODOC2.0 label.

            leftyl Lefty Leverenz added a comment - Doc note: This adds hive.vectorized.execution.reducesink.new.enabled to HiveConf.java, so it needs to be documented in the wiki for release 2.0.0. Configuration Properties – Query and DDL Execution Adding a TODOC2.0 label.
            leftyl Lefty Leverenz added a comment - - edited

            mmccline, an extra file snuck into the commit: HiveConf.java.orig (not in patch 6).

            leftyl Lefty Leverenz added a comment - - edited mmccline , an extra file snuck into the commit: HiveConf.java.orig (not in patch 6).
            mmccline Matt McCline added a comment -

            Committed to master.

            mmccline Matt McCline added a comment - Committed to master.
            mmccline Matt McCline added a comment -

            Failures are old and unrelated.

            mmccline Matt McCline added a comment - Failures are old and unrelated.
            hiveqa Hive QA added a comment -

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12769966/HIVE-12290.06.patch

            SUCCESS: +1 due to 3 test(s) being added or modified.

            ERROR: -1 due to 4 failed/errored test(s), 9756 tests executed
            Failed tests:

            org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
            org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
            org.apache.hadoop.hive.hwi.TestHWISessionManager.testHiveDriver
            org.apache.hive.jdbc.TestSSL.testSSLVersion
            

            Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5883/testReport
            Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5883/console
            Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5883/

            Messages:

            Executing org.apache.hive.ptest.execution.TestCheckPhase
            Executing org.apache.hive.ptest.execution.PrepPhase
            Executing org.apache.hive.ptest.execution.ExecutionPhase
            Executing org.apache.hive.ptest.execution.ReportingPhase
            Tests exited with: TestsFailedException: 4 tests failed
            

            This message is automatically generated.

            ATTACHMENT ID: 12769966 - PreCommit-HIVE-TRUNK-Build

            hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12769966/HIVE-12290.06.patch SUCCESS: +1 due to 3 test(s) being added or modified. ERROR: -1 due to 4 failed/errored test(s), 9756 tests executed Failed tests: org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.hwi.TestHWISessionManager.testHiveDriver org.apache.hive.jdbc.TestSSL.testSSLVersion Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5883/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5883/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5883/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed This message is automatically generated. ATTACHMENT ID: 12769966 - PreCommit-HIVE-TRUNK-Build
            mmccline Matt McCline added a comment -

            Fix issues and try again.

            mmccline Matt McCline added a comment - Fix issues and try again.
            hiveqa Hive QA added a comment -

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12769937/HIVE-12290.05.patch

            SUCCESS: +1 due to 3 test(s) being added or modified.

            ERROR: -1 due to 12 failed/errored test(s), 9758 tests executed
            Failed tests:

            org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
            org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_3
            org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_mapjoin
            org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_interval_mapjoin
            org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_join_nulls
            org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_nullsafe_join
            org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join3
            org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_ptf
            org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
            org.apache.hadoop.hive.hwi.TestHWISessionManager.testHiveDriver
            org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
            org.apache.hive.jdbc.TestSSL.testSSLVersion
            

            Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5878/testReport
            Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5878/console
            Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5878/

            Messages:

            Executing org.apache.hive.ptest.execution.TestCheckPhase
            Executing org.apache.hive.ptest.execution.PrepPhase
            Executing org.apache.hive.ptest.execution.ExecutionPhase
            Executing org.apache.hive.ptest.execution.ReportingPhase
            Tests exited with: TestsFailedException: 12 tests failed
            

            This message is automatically generated.

            ATTACHMENT ID: 12769937 - PreCommit-HIVE-TRUNK-Build

            hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12769937/HIVE-12290.05.patch SUCCESS: +1 due to 3 test(s) being added or modified. ERROR: -1 due to 12 failed/errored test(s), 9758 tests executed Failed tests: org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_3 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_mapjoin org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_interval_mapjoin org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_join_nulls org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_nullsafe_join org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join3 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_ptf org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.hwi.TestHWISessionManager.testHiveDriver org.apache.hive.jdbc.TestSSL.testSSLFetchHttp org.apache.hive.jdbc.TestSSL.testSSLVersion Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5878/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5878/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5878/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 12 tests failed This message is automatically generated. ATTACHMENT ID: 12769937 - PreCommit-HIVE-TRUNK-Build
            mmccline Matt McCline added a comment -

            Add missing Tez files from testconfiguration.properties

            mmccline Matt McCline added a comment - Add missing Tez files from testconfiguration.properties
            mmccline Matt McCline added a comment -

            Move HiveTableUtil to common.

            mmccline Matt McCline added a comment - Move HiveTableUtil to common.

            mmccline: code LGTM - +1 tests pending.

            Needs clarification on the extra "Optimized by CBO" change in the q.out (marked on RB).

            gopalv Gopal Vijayaraghavan added a comment - mmccline : code LGTM - +1 tests pending. Needs clarification on the extra "Optimized by CBO" change in the q.out (marked on RB).
            mmccline Matt McCline added a comment -

            gopalv Attached new patch with more review comment fixes.

            mmccline Matt McCline added a comment - gopalv Attached new patch with more review comment fixes.
            mmccline Matt McCline added a comment -

            And thanks for do a quick measurement. 25% reduction isn't bad.

            mmccline Matt McCline added a comment - And thanks for do a quick measurement. 25% reduction isn't bad.
            mmccline Matt McCline added a comment -

            gopalv Thank you for the review!

            Submitted new patch with review changes.

            mmccline Matt McCline added a comment - gopalv Thank you for the review! Submitted new patch with review changes.

            mmccline: comments on RB.

            This patch, after minor modifications took a 55 million row RS from 7.8s -> 5.8s.

            gopalv Gopal Vijayaraghavan added a comment - mmccline : comments on RB. This patch, after minor modifications took a 55 million row RS from 7.8s -> 5.8s.
            hiveqa Hive QA added a comment -

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12769504/HIVE-12290.01.patch

            SUCCESS: +1 due to 3 test(s) being added or modified.

            ERROR: -1 due to 7 failed/errored test(s), 9737 tests executed
            Failed tests:

            org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
            org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_index_bitmap3
            org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_stats_counter_partitioned
            org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
            org.apache.hadoop.hive.hwi.TestHWISessionManager.testHiveDriver
            org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
            org.apache.hive.jdbc.TestSSL.testSSLVersion
            

            Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5859/testReport
            Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5859/console
            Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5859/

            Messages:

            Executing org.apache.hive.ptest.execution.TestCheckPhase
            Executing org.apache.hive.ptest.execution.PrepPhase
            Executing org.apache.hive.ptest.execution.ExecutionPhase
            Executing org.apache.hive.ptest.execution.ReportingPhase
            Tests exited with: TestsFailedException: 7 tests failed
            

            This message is automatically generated.

            ATTACHMENT ID: 12769504 - PreCommit-HIVE-TRUNK-Build

            hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12769504/HIVE-12290.01.patch SUCCESS: +1 due to 3 test(s) being added or modified. ERROR: -1 due to 7 failed/errored test(s), 9737 tests executed Failed tests: org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_stats_counter_partitioned org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.hwi.TestHWISessionManager.testHiveDriver org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery org.apache.hive.jdbc.TestSSL.testSSLVersion Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5859/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5859/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5859/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed This message is automatically generated. ATTACHMENT ID: 12769504 - PreCommit-HIVE-TRUNK-Build

            People

              mmccline Matt McCline
              mmccline Matt McCline
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: