Hive
  1. Hive
  2. HIVE-4160 Vectorized Query Execution in Hive
  3. HIVE-4512

The vectorized plan is not picking right expression class for string concatenation.

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: vectorization-branch
    • Fix Version/s: vectorization-branch, 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      The vectorized plan is not picking right expression class for string concatenation.

      1. HIVE-4512.1-vectorization.patch
        8 kB
        Eric Hanson
      2. HIVE-4512.2-vectorization.patch
        8 kB
        Eric Hanson
      3. HIVE-4512.3-vectorization.patch
        8 kB
        Eric Hanson
      4. HIVE-4512.4-vectorization.patch
        9 kB
        Eric Hanson
      5. HIVE-4512.5-vectorization.patch
        9 kB
        Eric Hanson

        Activity

        Hide
        Ashutosh Chauhan added a comment -

        Committed to branch. Thanks, Eric!

        Show
        Ashutosh Chauhan added a comment - Committed to branch. Thanks, Eric!
        Hide
        Eric Hanson added a comment -

        Re-based patch after yesterday's merge from trunk to vectorization branch. Ran vectorized string function tests and ad-hoc end-to-end tests to verify operation of concat in vectorized mode.

        Show
        Eric Hanson added a comment - Re-based patch after yesterday's merge from trunk to vectorization branch. Ran vectorized string function tests and ad-hoc end-to-end tests to verify operation of concat in vectorized mode.
        Hide
        Eric Hanson added a comment -

        Rebased patch off most recent changes to add vectorized UDF adaptor (HIVE-4961). Also updated Vectorizer to reference Concat class and tested Concat function in end-to-end query runs. Re-ran junit tests for concat and they pass.

        Show
        Eric Hanson added a comment - Rebased patch off most recent changes to add vectorized UDF adaptor ( HIVE-4961 ). Also updated Vectorizer to reference Concat class and tested Concat function in end-to-end query runs. Re-ran junit tests for concat and they pass.
        Hide
        Eric Hanson added a comment -

        I'm re-basing the patch on the latest changes. Stay tuned.

        Show
        Eric Hanson added a comment - I'm re-basing the patch on the latest changes. Stay tuned.
        Hide
        Eric Hanson added a comment -

        These test failures are not related to this patch. See the discussion in https://issues.apache.org/jira/browse/HIVE-4961. That applies here too.

        Show
        Eric Hanson added a comment - These test failures are not related to this patch. See the discussion in https://issues.apache.org/jira/browse/HIVE-4961 . That applies here too.
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12603399/HIVE-4512.3-vectorization.patch

        ERROR: -1 due to 17 failed/errored test(s), 3951 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input4
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_plan_json
        org.apache.hadoop.hive.ql.io.orc.TestFileDump.testDictionaryThreshold
        org.apache.hadoop.hive.ql.io.orc.TestFileDump.testDump
        org.apache.hcatalog.api.TestHCatClient.testBasicDDLCommands
        org.apache.hcatalog.api.TestHCatClient.testPartitionsHCatClientImpl
        org.apache.hive.hcatalog.api.TestHCatClient.testBasicDDLCommands
        org.apache.hive.hcatalog.api.TestHCatClient.testDatabaseLocation
        org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSchema
        org.apache.hive.hcatalog.api.TestHCatClient.testPartitionsHCatClientImpl
        org.apache.hive.hcatalog.fileformats.TestOrcDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask
        org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTable
        org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask
        org.apache.hive.hcatalog.mapreduce.TestHCatExternalPartitioned.testHCatPartitionedTable
        org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes
        org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic
        org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
        

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/768/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/768/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests failed with: TestsFailedException: 17 tests failed
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12603399/HIVE-4512.3-vectorization.patch ERROR: -1 due to 17 failed/errored test(s), 3951 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_plan_json org.apache.hadoop.hive.ql.io.orc.TestFileDump.testDictionaryThreshold org.apache.hadoop.hive.ql.io.orc.TestFileDump.testDump org.apache.hcatalog.api.TestHCatClient.testBasicDDLCommands org.apache.hcatalog.api.TestHCatClient.testPartitionsHCatClientImpl org.apache.hive.hcatalog.api.TestHCatClient.testBasicDDLCommands org.apache.hive.hcatalog.api.TestHCatClient.testDatabaseLocation org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSchema org.apache.hive.hcatalog.api.TestHCatClient.testPartitionsHCatClientImpl org.apache.hive.hcatalog.fileformats.TestOrcDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTable org.apache.hive.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask org.apache.hive.hcatalog.mapreduce.TestHCatExternalPartitioned.testHCatPartitionedTable org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/768/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/768/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 17 tests failed This message is automatically generated.
        Hide
        Eric Hanson added a comment -

        Based patch off the latest vectorization branch

        Show
        Eric Hanson added a comment - Based patch off the latest vectorization branch
        Hide
        Hive QA added a comment -

        Overall: -1 no tests executed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12594218/HIVE-4512.2-vectorization.patch

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/193/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/193/console

        Messages:

        Executing org.apache.hive.ptest.execution.CleanupPhase
        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Tests failed with: IllegalStateException: Too many bad hosts: 0.6% (6 / 10) is greater than threshold of 50%
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12594218/HIVE-4512.2-vectorization.patch Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/193/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/193/console Messages: Executing org.apache.hive.ptest.execution.CleanupPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Tests failed with: IllegalStateException: Too many bad hosts: 0.6% (6 / 10) is greater than threshold of 50% This message is automatically generated.
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12594047/HIVE-4512.1-vectorization.patch

        ERROR: -1 due to 2 failed/errored test(s), 3410 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_view_cast
        

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/177/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/177/console

        Messages:

        Executing org.apache.hive.ptest.execution.CleanupPhase
        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests failed with: TestsFailedException: 2 tests failed
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12594047/HIVE-4512.1-vectorization.patch ERROR: -1 due to 2 failed/errored test(s), 3410 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_view_cast Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/177/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/177/console Messages: Executing org.apache.hive.ptest.execution.CleanupPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 2 tests failed This message is automatically generated.
        Hide
        Eric Hanson added a comment -

        Code review available at: https://reviews.apache.org/r/12926/

        Show
        Eric Hanson added a comment - Code review available at: https://reviews.apache.org/r/12926/
        Hide
        Eric Hanson added a comment -

        Implemented support in VectorizationContext.java for concat(col, scalar). Still need to finish scalar-col and col-col support.

        Show
        Eric Hanson added a comment - Implemented support in VectorizationContext.java for concat(col, scalar). Still need to finish scalar-col and col-col support.

          People

          • Assignee:
            Eric Hanson
            Reporter:
            Jitendra Nath Pandey
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development