Hive
  1. Hive
  2. HIVE-1056

Predicate push down does not work with UDTF's

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.6.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Predicate push down does not work with UDTF's in lateral views

      
      hive> SELECT * FROM src LATERAL VIEW explode(array(1,2,3)) myTable AS k WHERE k=1;
      FAILED: Unknown exception: null
      hive>
      
      
      1. HIVE-1056-0.6.patch.txt
        83 kB
        Carl Steinbach
      2. HIVE-1056.4.patch
        82 kB
        Paul Yang
      3. HIVE-1056.3.patch
        81 kB
        Paul Yang
      4. HIVE-1056.2.patch
        81 kB
        Paul Yang
      5. HIVE-1056.1.patch
        78 kB
        Paul Yang

        Activity

        Hide
        Paul Yang added a comment -
        • Enables UDTF's to emit rows on close. Although this was disabled initially, it should be allowed. Without this, the test with lateral views on partitioned tables will fail.
        • Enables PPD for LV. To make pruning easier, a new operator (LateralViewForward) is placed at the head of every LV DAG. Otherwise, it is difficult for the pruner to handle the special case for LV's.
        Show
        Paul Yang added a comment - Enables UDTF's to emit rows on close. Although this was disabled initially, it should be allowed. Without this, the test with lateral views on partitioned tables will fail. Enables PPD for LV. To make pruning easier, a new operator (LateralViewForward) is placed at the head of every LV DAG. Otherwise, it is difficult for the pruner to handle the special case for LV's.
        Hide
        Ning Zhang added a comment -

        Will take a look.

        Show
        Ning Zhang added a comment - Will take a look.
        Hide
        HBase Review Board added a comment -

        Message from: "Carl Steinbach" <carl@cloudera.com>

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        http://review.hbase.org/r/198/
        -----------------------------------------------------------

        Review request for Hive Developers.

        Summary
        -------

        Review for patch https://issues.apache.org/jira/secure/attachment/12441230/HIVE-1056.1.patch

        This addresses bug HIVE-1056.
        http://issues.apache.org/jira/browse/HIVE-1056

        Diffs


        contrib/src/java/org/apache/hadoop/hive/contrib/udtf/example/GenericUDTFCount2.java PRE-CREATION
        contrib/src/test/queries/clientpositive/udtf_output_on_close.q PRE-CREATION
        contrib/src/test/results/clientpositive/udtf_output_on_close.q.out PRE-CREATION
        ql/if/queryplan.thrift d387e8e
        ql/src/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java 99d1f2d
        ql/src/gen-php/queryplan_types.php 334b4f8
        ql/src/gen-py/queryplan/ttypes.py d228e68
        ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewForwardOperator.java PRE-CREATION
        ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewJoinOperator.java 371a7ac
        ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 03bd0bb
        ql/src/java/org/apache/hadoop/hive/ql/exec/UDTFOperator.java cab3057
        ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java a2f9dba
        ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewForwardDesc.java PRE-CREATION
        ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 6e1dfe4
        ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicatePushDown.java 19fe5f4
        ql/src/test/queries/clientpositive/lateral_view_ppd.q PRE-CREATION
        ql/src/test/results/clientpositive/lateral_view.q.out 8931455
        ql/src/test/results/clientpositive/lateral_view_ppd.q.out PRE-CREATION

        Diff: http://review.hbase.org/r/198/diff

        Testing
        -------

        Thanks,

        Carl

        Show
        HBase Review Board added a comment - Message from: "Carl Steinbach" <carl@cloudera.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/198/ ----------------------------------------------------------- Review request for Hive Developers. Summary ------- Review for patch https://issues.apache.org/jira/secure/attachment/12441230/HIVE-1056.1.patch This addresses bug HIVE-1056 . http://issues.apache.org/jira/browse/HIVE-1056 Diffs contrib/src/java/org/apache/hadoop/hive/contrib/udtf/example/GenericUDTFCount2.java PRE-CREATION contrib/src/test/queries/clientpositive/udtf_output_on_close.q PRE-CREATION contrib/src/test/results/clientpositive/udtf_output_on_close.q.out PRE-CREATION ql/if/queryplan.thrift d387e8e ql/src/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java 99d1f2d ql/src/gen-php/queryplan_types.php 334b4f8 ql/src/gen-py/queryplan/ttypes.py d228e68 ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewForwardOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewJoinOperator.java 371a7ac ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 03bd0bb ql/src/java/org/apache/hadoop/hive/ql/exec/UDTFOperator.java cab3057 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java a2f9dba ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewForwardDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 6e1dfe4 ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicatePushDown.java 19fe5f4 ql/src/test/queries/clientpositive/lateral_view_ppd.q PRE-CREATION ql/src/test/results/clientpositive/lateral_view.q.out 8931455 ql/src/test/results/clientpositive/lateral_view_ppd.q.out PRE-CREATION Diff: http://review.hbase.org/r/198/diff Testing ------- Thanks, Carl
        Hide
        Paul Yang added a comment -

        Regenerated patch.

        Show
        Paul Yang added a comment - Regenerated patch.
        Hide
        Paul Yang added a comment -

        Regenerated again.

        Show
        Paul Yang added a comment - Regenerated again.
        Hide
        Paul Yang added a comment -

        Added comment for LVF PPD rule.

        Show
        Paul Yang added a comment - Added comment for LVF PPD rule.
        Hide
        He Yongqiang added a comment -

        running tests now.

        Show
        He Yongqiang added a comment - running tests now.
        Hide
        He Yongqiang added a comment -

        Committed! Thanks Paul!

        Show
        He Yongqiang added a comment - Committed! Thanks Paul!
        Hide
        Carl Steinbach added a comment -

        We should backport this to 0.6

        Show
        Carl Steinbach added a comment - We should backport this to 0.6
        Hide
        Carl Steinbach added a comment -

        Attaching patch for branch-0.6

        Show
        Carl Steinbach added a comment - Attaching patch for branch-0.6
        Hide
        He Yongqiang added a comment -

        Committed to branch 0.6 together with HIVE-1452. Thanks Paul and Carl!

        Show
        He Yongqiang added a comment - Committed to branch 0.6 together with HIVE-1452 . Thanks Paul and Carl!
        Hide
        He Yongqiang added a comment -
        Show
        He Yongqiang added a comment - typo: HIVE-1452 -> HIVE-1455

          People

          • Assignee:
            Paul Yang
            Reporter:
            Paul Yang
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development