Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0, 0.14.0
    • Fix Version/s: 0.13.0, 0.14.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      HIVE-6455 added support for scalable dynamic partitioning. This is subtask to make HIVE-6455 work with vectorized operators.

      1. HIVE-6735.1.patch
        561 kB
        Prasanth Jayachandran
      2. HIVE-6735.2.patch
        561 kB
        Prasanth Jayachandran
      3. HIVE-6735.2.patch
        561 kB
        Prasanth Jayachandran
      4. HIVE-6735.3.patch
        560 kB
        Prasanth Jayachandran

        Issue Links

          Activity

          Hide
          Prasanth Jayachandran added a comment -

          Only VectorReduceSink needs changes to make it work in vectorized mode. Changes are similar to HIVE-6455. Added qfile test for MR and Tez for vectorized mode.

          Show
          Prasanth Jayachandran added a comment - Only VectorReduceSink needs changes to make it work in vectorized mode. Changes are similar to HIVE-6455 . Added qfile test for MR and Tez for vectorized mode.
          Hide
          Gopal V added a comment -

          Testing this on a scale=200 data-load.

          Show
          Gopal V added a comment - Testing this on a scale=200 data-load.
          Hide
          Jitendra Nath Pandey added a comment -

          +1 LGTM

          Show
          Jitendra Nath Pandey added a comment - +1 LGTM
          Hide
          Prasanth Jayachandran added a comment -

          Addressed Gopal V's comments.

          Show
          Prasanth Jayachandran added a comment - Addressed Gopal V 's comments.
          Hide
          Prasanth Jayachandran added a comment -

          Reuploading for jenkins

          Show
          Prasanth Jayachandran added a comment - Reuploading for jenkins
          Hide
          Prasanth Jayachandran added a comment -

          Harish Butani this is a sub-task of feature HIVE-6455. Can we have this in 0.13 as well?

          Show
          Prasanth Jayachandran added a comment - Harish Butani this is a sub-task of feature HIVE-6455 . Can we have this in 0.13 as well?
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12636757/HIVE-6735.2.patch

          ERROR: -1 due to 2 failed/errored test(s), 5448 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
          org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1958/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1958/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12636757

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12636757/HIVE-6735.2.patch ERROR: -1 due to 2 failed/errored test(s), 5448 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1958/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1958/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12636757
          Hide
          Prasanth Jayachandran added a comment -

          I ran bucket_num_reducers.q locally and it ran successfully. Also that test is not related as it doesn't use vectorization or dynamic partitioning. The failures are unrelated.

          Show
          Prasanth Jayachandran added a comment - I ran bucket_num_reducers.q locally and it ran successfully. Also that test is not related as it doesn't use vectorization or dynamic partitioning. The failures are unrelated.
          Hide
          Harish Butani added a comment -

          +1 for 0.13

          Show
          Harish Butani added a comment - +1 for 0.13
          Hide
          Gopal V added a comment -

          The updated .2 patch runs in 859 seconds, where the previous one ran in ~350 seconds.

          The code needs to loop through all partition columns when generating hashCode - I got 13 unique hashCodes for 78 different keys, with .1 patch I was getting 78 different hashCodes.

          Show
          Gopal V added a comment - The updated .2 patch runs in 859 seconds, where the previous one ran in ~350 seconds. The code needs to loop through all partition columns when generating hashCode - I got 13 unique hashCodes for 78 different keys, with .1 patch I was getting 78 different hashCodes.
          Hide
          Prasanth Jayachandran added a comment -

          Gopal V the reason for the lesser hashcode generation is mentioned in HIVE-6761. I will fix this patch to include all partition columns.

          Show
          Prasanth Jayachandran added a comment - Gopal V the reason for the lesser hashcode generation is mentioned in HIVE-6761 . I will fix this patch to include all partition columns.
          Hide
          Prasanth Jayachandran added a comment -

          Same as .1 patch with some redundant code removed.

          Show
          Prasanth Jayachandran added a comment - Same as .1 patch with some redundant code removed.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12637049/HIVE-6735.3.patch

          ERROR: -1 due to 2 failed/errored test(s), 5492 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1986/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1986/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12637049

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12637049/HIVE-6735.3.patch ERROR: -1 due to 2 failed/errored test(s), 5492 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1986/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1986/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12637049
          Hide
          Prasanth Jayachandran added a comment -

          The test failures are unrelated. They pass locally on my system.

          Show
          Prasanth Jayachandran added a comment - The test failures are unrelated. They pass locally on my system.
          Hide
          Harish Butani added a comment -

          +1 for 0.13

          Show
          Harish Butani added a comment - +1 for 0.13
          Hide
          Gunther Hagleitner added a comment -

          Committed to trunk and branch. Thanks Prasanth Jayachandran!

          Show
          Gunther Hagleitner added a comment - Committed to trunk and branch. Thanks Prasanth Jayachandran !
          Hide
          Thejas M Nair added a comment -

          This has been fixed in 0.14 release. Please open new jira if you see any issues.

          Show
          Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.

            People

            • Assignee:
              Prasanth Jayachandran
              Reporter:
              Prasanth Jayachandran
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development