Hive
  1. Hive
  2. HIVE-4878

With Dynamic partitioning, some queries would scan default partition even if query is not using it.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.0
    • Fix Version/s: 0.12.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      With Dynamic partitioning, Hive would scan default partitions in some cases even if query excludes it.

      As part of partition pruning, predicate is narrowed down to those pieces that involve partition columns only. This predicate is then evaluated with partition values to determine, if scan should include those partitions.

      But in some cases (like when comparing "_HIVE_DEFAULT_PARTITION_" to numeric data types) expression evaluation would fail and would return NULL instead of true/false. In such cases the partition is added to unknown partitions which is then subsequently scanned.

      1. HIVE-4878.patch
        23 kB
        Laljo John Pullokkaran

        Issue Links

          Activity

          Hide
          Ashutosh Chauhan added a comment -

          This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

          Show
          Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, John!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, John!
          Hide
          Ashutosh Chauhan added a comment -

          +1

          Show
          Ashutosh Chauhan added a comment - +1
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12594014/HIVE-4878.patch

          SUCCESS: +1 2652 tests passed

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/175/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/175/console

          Messages:

          Executing org.apache.hive.ptest.execution.CleanupPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12594014/HIVE-4878.patch SUCCESS: +1 2652 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/175/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/175/console Messages: Executing org.apache.hive.ptest.execution.CleanupPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
          Hide
          Laljo John Pullokkaran added a comment -

          Resubmitted the patch to do partition skip regardless of "Dynamic Partition Mode".

          This fix avoids scanning default partition if all of the following is true:
          a) partition pruning expression failed to evaluate for a given partition.
          b) at the least one of the columns in the partition is default partition.

          Show
          Laljo John Pullokkaran added a comment - Resubmitted the patch to do partition skip regardless of "Dynamic Partition Mode". This fix avoids scanning default partition if all of the following is true: a) partition pruning expression failed to evaluate for a given partition. b) at the least one of the columns in the partition is default partition.
          Hide
          Ashutosh Chauhan added a comment -

          Comment on RB.

          Show
          Ashutosh Chauhan added a comment - Comment on RB.
          Hide
          Laljo John Pullokkaran added a comment -

          Hive Unit Tests passed.

          Show
          Laljo John Pullokkaran added a comment - Hive Unit Tests passed.
          Hide
          Laljo John Pullokkaran added a comment -

          This fix avoids scanning default partition in "strict mode"if all of the following is true:
          a) partition pruning expression failed to evaluate for a given partition.
          b) at the least one of the columns in the partition is default partition.

          Show
          Laljo John Pullokkaran added a comment - This fix avoids scanning default partition in "strict mode"if all of the following is true: a) partition pruning expression failed to evaluate for a given partition. b) at the least one of the columns in the partition is default partition.

            People

            • Assignee:
              Laljo John Pullokkaran
              Reporter:
              Laljo John Pullokkaran
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development