Hive
  1. Hive
  2. HIVE-6455 Scalable dynamic partitioning and bucketing optimization
  3. HIVE-6761

Hashcode computation does not use maximum parallelism for scalable dynamic partitioning

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0, 0.14.0
    • Fix Version/s: 0.13.0, 0.14.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      Hashcode computation for HIVE-6455 should consider all the partitioning columns and bucket number to distribute the rows. The following code

      for (int i = 0; i < partitionEval.length - 1; i++) {
      

      ignores the last partition column thereby generating lesser hashcodes.

      1. HIVE-6761.1.patch
        2 kB
        Prasanth Jayachandran

        Activity

        Show
        Prasanth Jayachandran added a comment - This is the reason of lesser hashcodes generation as mentioned by Gopal V here https://issues.apache.org/jira/browse/HIVE-6735?focusedCommentId=13948046&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13948046
        Hide
        Gunther Hagleitner added a comment -

        Looks good to me. +1

        Show
        Gunther Hagleitner added a comment - Looks good to me. +1
        Hide
        Alan Gates added a comment -

        Ran tests locally, all looks good.

        Show
        Alan Gates added a comment - Ran tests locally, all looks good.
        Hide
        Prasanth Jayachandran added a comment -

        Thanks Alan Gates.

        Show
        Prasanth Jayachandran added a comment - Thanks Alan Gates .
        Hide
        Harish Butani added a comment -

        +1 for 0.13

        Show
        Harish Butani added a comment - +1 for 0.13
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12637047/HIVE-6761.1.patch

        ERROR: -1 due to 2 failed/errored test(s), 5492 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1989/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1989/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 2 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12637047

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12637047/HIVE-6761.1.patch ERROR: -1 due to 2 failed/errored test(s), 5492 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1989/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1989/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12637047
        Hide
        Gunther Hagleitner added a comment -

        Committed to trunk and branch. Thanks Prasanth Jayachandran!

        Show
        Gunther Hagleitner added a comment - Committed to trunk and branch. Thanks Prasanth Jayachandran !
        Hide
        Thejas M Nair added a comment -

        This has been fixed in 0.14 release. Please open new jira if you see any issues.

        Show
        Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.

          People

          • Assignee:
            Prasanth Jayachandran
            Reporter:
            Prasanth Jayachandran
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development