Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-10106

Regression : Dynamic partition pruning not working after HIVE-9976

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0
    • Component/s: Hive
    • Labels:
      None

      Description

      After HIVE-9976 got checked in dynamic partition pruning doesn't work.
      Partitions are pruned and later show up in splits.

      1. HIVE-10106.1.patch
        7 kB
        Siddharth Seth

        Activity

        Hide
        sseth Siddharth Seth added a comment -

        I believe this is caused by the way mapWork is setup. It's now created in the constructor of HiveSplitGenerator. The constructor and the initialize method may not be invoked in the same thread. As a result, the initialize method ends up seing a different copy of mapWork from the one modified in the pruner.
        Attaching a patch to fix this - by setting the mapWork in the initialize method.
        Gunther Hagleitner - please review, and validate the theory.
        Mostafa Mokhtar - I wasn't able to reproduce this. Seing pruning work as it should for the simple query that you'd sent me offline. May need help reproducing the issue and validating the patch. Thanks

        Show
        sseth Siddharth Seth added a comment - I believe this is caused by the way mapWork is setup. It's now created in the constructor of HiveSplitGenerator. The constructor and the initialize method may not be invoked in the same thread. As a result, the initialize method ends up seing a different copy of mapWork from the one modified in the pruner. Attaching a patch to fix this - by setting the mapWork in the initialize method. Gunther Hagleitner - please review, and validate the theory. Mostafa Mokhtar - I wasn't able to reproduce this. Seing pruning work as it should for the simple query that you'd sent me offline. May need help reproducing the issue and validating the patch. Thanks
        Hide
        hagleitn Gunther Hagleitner added a comment -

        That sounds correct to me. +1.

        Show
        hagleitn Gunther Hagleitner added a comment - That sounds correct to me. +1.
        Hide
        hiveqa Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12707704/HIVE-10106.1.patch

        ERROR: -1 due to 5 failed/errored test(s), 8677 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_remote_script
        org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_root_dir_external_table
        org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_scriptfile1
        org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testMetastoreProxyUser
        org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3186/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3186/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3186/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 5 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12707704 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12707704/HIVE-10106.1.patch ERROR: -1 due to 5 failed/errored test(s), 8677 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_remote_script org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_scriptfile1 org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testMetastoreProxyUser org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3186/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3186/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3186/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed This message is automatically generated. ATTACHMENT ID: 12707704 - PreCommit-HIVE-TRUNK-Build
        Hide
        hagleitn Gunther Hagleitner added a comment -

        Committed to trunk.

        Show
        hagleitn Gunther Hagleitner added a comment - Committed to trunk.
        Hide
        sushanth Sushanth Sowmyan added a comment -

        This issue has been fixed and released as part of the 1.2.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

        Show
        sushanth Sushanth Sowmyan added a comment - This issue has been fixed and released as part of the 1.2.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          People

          • Assignee:
            sseth Siddharth Seth
            Reporter:
            mmokhtar Mostafa Mokhtar
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development