Hive
  1. Hive
  2. HIVE-2451

TABLESAMBLE(BUCKET xxx) sometimes doesn't trigger input pruning as regression of HIVE-1538

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      Example:

      select count(1) from <bucket_table> TABLESAMPLE(BUCKET xxx out of yyy) where <partition_column> = 'xxx'

      will not trigger input pruning.

      The reason is that we assume sample filtering operator only happens as the second filter after table scan, which is broken by HIVE-1538, even if the feature doesn't turn on.

      1. HIVE-2451.1.patch
        0.7 kB
        Siying Dong
      2. HIVE-2451.2.patch
        97 kB
        Siying Dong
      3. HIVE-2451.3.patch
        137 kB
        Siying Dong

        Activity

        Hide
        Siying Dong added a comment -

        Fix the problem by considering sample filter operator can be the first filter operator after table scan.

        Show
        Siying Dong added a comment - Fix the problem by considering sample filter operator can be the first filter operator after table scan.
        Hide
        He Yongqiang added a comment -

        +1, will commit after tests pass

        Show
        He Yongqiang added a comment - +1, will commit after tests pass
        Hide
        Siying Dong added a comment -

        There's a bug.

        Show
        Siying Dong added a comment - There's a bug.
        Hide
        Siying Dong added a comment -

        Changed an assert issue and recover the some test result files which were changed incorrectly by HIVE-1538.

        Show
        Siying Dong added a comment - Changed an assert issue and recover the some test result files which were changed incorrectly by HIVE-1538 .
        Hide
        Siying Dong added a comment -

        Reran all test suites and fixed another several wrong test results.

        Show
        Siying Dong added a comment - Reran all test suites and fixed another several wrong test results.
        Hide
        He Yongqiang added a comment -

        committed, thanks Siying!

        Show
        He Yongqiang added a comment - committed, thanks Siying!
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #965 (See https://builds.apache.org/job/Hive-trunk-h0.21/965/)
        HIVE-2451: TABLESAMBLE(BUCKET xxx) sometimes doesn't trigger input pruning (Siying Dong via He Yongqiang)

        heyongqiang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172891
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SamplePruner.java
        • /hive/trunk/ql/src/test/results/clientpositive/sample10.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/sample2.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/sample4.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/sample6.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/sample9.q.out
        • /hive/trunk/ql/src/test/results/compiler/plan/sample2.q.xml
        • /hive/trunk/ql/src/test/results/compiler/plan/sample4.q.xml
        • /hive/trunk/ql/src/test/results/compiler/plan/sample6.q.xml
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #965 (See https://builds.apache.org/job/Hive-trunk-h0.21/965/ ) HIVE-2451 : TABLESAMBLE(BUCKET xxx) sometimes doesn't trigger input pruning (Siying Dong via He Yongqiang) heyongqiang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172891 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SamplePruner.java /hive/trunk/ql/src/test/results/clientpositive/sample10.q.out /hive/trunk/ql/src/test/results/clientpositive/sample2.q.out /hive/trunk/ql/src/test/results/clientpositive/sample4.q.out /hive/trunk/ql/src/test/results/clientpositive/sample6.q.out /hive/trunk/ql/src/test/results/clientpositive/sample9.q.out /hive/trunk/ql/src/test/results/compiler/plan/sample2.q.xml /hive/trunk/ql/src/test/results/compiler/plan/sample4.q.xml /hive/trunk/ql/src/test/results/compiler/plan/sample6.q.xml

          People

          • Assignee:
            Siying Dong
            Reporter:
            Siying Dong
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development