Description
Currently, VectorReduceSinkOperator is a pass-thru to ReduceSinkOperator, so we incur object inspector costs.
Native vectorization will not use object inspectors and allocate memory up front that will be reused for each batch.
Attachments
Attachments
- HIVE-12290.01.patch
- 499 kB
- Matt McCline
- HIVE-12290.02.patch
- 501 kB
- Matt McCline
- HIVE-12290.03.patch
- 522 kB
- Matt McCline
- HIVE-12290.04.patch
- 522 kB
- Matt McCline
- HIVE-12290.05.patch
- 524 kB
- Matt McCline
- HIVE-12290.06.patch
- 524 kB
- Matt McCline
Issue Links
- relates to
-
HIVE-23852 Natively support Date type in ReduceSink operator
- Closed
Activity
Matt reverted the addition of HiveConf.java.orig with commit 9ba2cdfd9a8eae371048de3fa92131517d8f8d6d.
Doc note: This adds hive.vectorized.execution.reducesink.new.enabled to HiveConf.java, so it needs to be documented in the wiki for release 2.0.0.
Adding a TODOC2.0 label.
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12769966/HIVE-12290.06.patch
SUCCESS: +1 due to 3 test(s) being added or modified.
ERROR: -1 due to 4 failed/errored test(s), 9756 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.hwi.TestHWISessionManager.testHiveDriver org.apache.hive.jdbc.TestSSL.testSSLVersion
Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5883/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5883/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5883/
Messages:
Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed
This message is automatically generated.
ATTACHMENT ID: 12769966 - PreCommit-HIVE-TRUNK-Build
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12769937/HIVE-12290.05.patch
SUCCESS: +1 due to 3 test(s) being added or modified.
ERROR: -1 due to 12 failed/errored test(s), 9758 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_3 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_mapjoin org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_interval_mapjoin org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_join_nulls org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_nullsafe_join org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join3 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_ptf org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.hwi.TestHWISessionManager.testHiveDriver org.apache.hive.jdbc.TestSSL.testSSLFetchHttp org.apache.hive.jdbc.TestSSL.testSSLVersion
Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5878/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5878/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5878/
Messages:
Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 12 tests failed
This message is automatically generated.
ATTACHMENT ID: 12769937 - PreCommit-HIVE-TRUNK-Build
mmccline: code LGTM - +1 tests pending.
Needs clarification on the extra "Optimized by CBO" change in the q.out (marked on RB).
gopalv Thank you for the review!
Submitted new patch with review changes.
mmccline: comments on RB.
This patch, after minor modifications took a 55 million row RS from 7.8s -> 5.8s.
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12769504/HIVE-12290.01.patch
SUCCESS: +1 due to 3 test(s) being added or modified.
ERROR: -1 due to 7 failed/errored test(s), 9737 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_stats_counter_partitioned org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.hwi.TestHWISessionManager.testHiveDriver org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery org.apache.hive.jdbc.TestSSL.testSSLVersion
Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5859/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5859/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5859/
Messages:
Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed
This message is automatically generated.
ATTACHMENT ID: 12769504 - PreCommit-HIVE-TRUNK-Build
Doc done: