[HIVE-4512] The vectorized plan is not picking right expression class for string concatenation. - ASF JIRA

Details

Type: Sub-task
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: vectorization-branch
Fix Version/s: vectorization-branch, 0.13.0
Component/s: None
Labels:
None

Description

The vectorized plan is not picking right expression class for string concatenation.

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending

Attachments

HIVE-4512.1-vectorization.patch

24/Jul/13 23:23

8 kB

Eric Hanson
HIVE-4512.2-vectorization.patch

25/Jul/13 19:11

8 kB

Eric Hanson
HIVE-4512.3-vectorization.patch

16/Sep/13 20:02

8 kB

Eric Hanson
HIVE-4512.4-vectorization.patch

18/Sep/13 01:40

9 kB

Eric Hanson
HIVE-4512.5-vectorization.patch

20/Sep/13 17:58

9 kB

Eric Hanson

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Eric Hanson added a comment - 24/Jul/13 02:32

Implemented support in VectorizationContext.java for concat(col, scalar). Still need to finish scalar-col and col-col support.

Show

Eric Hanson added a comment - 24/Jul/13 02:32 Implemented support in VectorizationContext.java for concat(col, scalar). Still need to finish scalar-col and col-col support.

Hide

Permalink

Eric Hanson added a comment - 24/Jul/13 23:34

Code review available at: https://reviews.apache.org/r/12926/

Show

Eric Hanson added a comment - 24/Jul/13 23:34 Code review available at: https://reviews.apache.org/r/12926/

Hide

Permalink

Hive QA added a comment - 25/Jul/13 09:23

Overall: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12594047/HIVE-4512.1-vectorization.patch

ERROR: -1 due to 2 failed/errored test(s), 3410 tests executed
Failed tests:

org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_view_cast

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/177/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/177/console

Messages:

Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 2 tests failed

This message is automatically generated.

Show

Hive QA added a comment - 25/Jul/13 09:23 Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12594047/HIVE-4512.1-vectorization.patch ERROR: -1 due to 2 failed/errored test(s), 3410 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_view_cast Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/177/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/177/console Messages: Executing org.apache.hive.ptest.execution.CleanupPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 2 tests failed This message is automatically generated.

Hide

Permalink

Hive QA added a comment - 26/Jul/13 17:43

Overall: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12594218/HIVE-4512.2-vectorization.patch

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/193/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/193/console

Messages:

Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Tests failed with: IllegalStateException: Too many bad hosts: 0.6% (6 / 10) is greater than threshold of 50%

This message is automatically generated.

Show

Hive QA added a comment - 26/Jul/13 17:43 Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12594218/HIVE-4512.2-vectorization.patch Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/193/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/193/console Messages: Executing org.apache.hive.ptest.execution.CleanupPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Tests failed with: IllegalStateException: Too many bad hosts: 0.6% (6 / 10) is greater than threshold of 50% This message is automatically generated.

Hide

Permalink

Eric Hanson added a comment - 16/Sep/13 20:02

Based patch off the latest vectorization branch

Show

Eric Hanson added a comment - 16/Sep/13 20:02 Based patch off the latest vectorization branch

Hide

Permalink

Hive QA added a comment - 17/Sep/13 02:08

Overall: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12603399/HIVE-4512.3-vectorization.patch

ERROR: -1 due to 17 failed/errored test(s), 3951 tests executed
Failed tests:

org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_plan_json
org.apache.hadoop.hive.ql.io.orc.TestFileDump.testDictionaryThreshold
org.apache.hadoop.hive.ql.io.orc.TestFileDump.testDump
org.apache.hcatalog.api.TestHCatClient.testBasicDDLCommands
org.apache.hcatalog.api.TestHCatClient.testPartitionsHCatClientImpl
org.apache.hive.hcatalog.api.TestHCatClient.testBasicDDLCommands
org.apache.hive.hcatalog.api.TestHCatClient.testDatabaseLocation
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSchema
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionsHCatClientImpl
org.apache.hive.hcatalog.fileformats.TestOrcDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTable
org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask
org.apache.hive.hcatalog.mapreduce.TestHCatExternalPartitioned.testHCatPartitionedTable
org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes
org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/768/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/768/console

Messages:

Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 17 tests failed

This message is automatically generated.

Show

Hive QA added a comment - 17/Sep/13 02:08 Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12603399/HIVE-4512.3-vectorization.patch ERROR: -1 due to 17 failed/errored test(s), 3951 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_plan_json org.apache.hadoop.hive.ql.io.orc.TestFileDump.testDictionaryThreshold org.apache.hadoop.hive.ql.io.orc.TestFileDump.testDump org.apache.hcatalog.api.TestHCatClient.testBasicDDLCommands org.apache.hcatalog.api.TestHCatClient.testPartitionsHCatClientImpl org.apache.hive.hcatalog.api.TestHCatClient.testBasicDDLCommands org.apache.hive.hcatalog.api.TestHCatClient.testDatabaseLocation org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSchema org.apache.hive.hcatalog.api.TestHCatClient.testPartitionsHCatClientImpl org.apache.hive.hcatalog.fileformats.TestOrcDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTable org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask org.apache.hive.hcatalog.mapreduce.TestHCatExternalPartitioned.testHCatPartitionedTable org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/768/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/768/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 17 tests failed This message is automatically generated.

Hide

Permalink

Eric Hanson added a comment - 17/Sep/13 19:59

These test failures are not related to this patch. See the discussion in https://issues.apache.org/jira/browse/HIVE-4961. That applies here too.

Show

Eric Hanson added a comment - 17/Sep/13 19:59 These test failures are not related to this patch. See the discussion in https://issues.apache.org/jira/browse/HIVE-4961 . That applies here too.

Hide

Permalink

Eric Hanson added a comment - 18/Sep/13 00:56

I'm re-basing the patch on the latest changes. Stay tuned.

Show

Eric Hanson added a comment - 18/Sep/13 00:56 I'm re-basing the patch on the latest changes. Stay tuned.

Hide

Permalink

Eric Hanson added a comment - 18/Sep/13 01:40

Rebased patch off most recent changes to add vectorized UDF adaptor (~~HIVE-4961~~). Also updated Vectorizer to reference Concat class and tested Concat function in end-to-end query runs. Re-ran junit tests for concat and they pass.

Show

Eric Hanson added a comment - 18/Sep/13 01:40 Rebased patch off most recent changes to add vectorized UDF adaptor ( HIVE-4961 ). Also updated Vectorizer to reference Concat class and tested Concat function in end-to-end query runs. Re-ran junit tests for concat and they pass.

Hide

Permalink

Eric Hanson added a comment - 20/Sep/13 17:58

Re-based patch after yesterday's merge from trunk to vectorization branch. Ran vectorized string function tests and ad-hoc end-to-end tests to verify operation of concat in vectorized mode.

Show

Eric Hanson added a comment - 20/Sep/13 17:58 Re-based patch after yesterday's merge from trunk to vectorization branch. Ran vectorized string function tests and ad-hoc end-to-end tests to verify operation of concat in vectorized mode.

Hide

Permalink

Ashutosh Chauhan added a comment - 20/Sep/13 18:14

Committed to branch. Thanks, Eric!

Show

Ashutosh Chauhan added a comment - 20/Sep/13 18:14 Committed to branch. Thanks, Eric!

People

Assignee:

Eric Hanson

Reporter:

Jitendra Nath Pandey

Votes:

0 Vote for this issue

Watchers:

3 Start watching this issue

Dates

Created:

07/May/13 00:23

Updated:

23/Oct/13 22:59

Resolved:

20/Sep/13 18:14

Development

Agile

View on Board