Hive
  1. Hive
  2. HIVE-1056

Predicate push down does not work with UDTF's

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.6.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Predicate push down does not work with UDTF's in lateral views

      
      hive> SELECT * FROM src LATERAL VIEW explode(array(1,2,3)) myTable AS k WHERE k=1;
      FAILED: Unknown exception: null
      hive>
      
      
      1. HIVE-1056.1.patch
        78 kB
        Paul Yang
      2. HIVE-1056.2.patch
        81 kB
        Paul Yang
      3. HIVE-1056.3.patch
        81 kB
        Paul Yang
      4. HIVE-1056.4.patch
        82 kB
        Paul Yang
      5. HIVE-1056-0.6.patch.txt
        83 kB
        Carl Steinbach

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        82d 21h 51m 1 Paul Yang 09/Apr/10 00:24
        Resolved Resolved Reopened Reopened
        8d 16h 56m 1 Carl Steinbach 22/Jul/10 20:26
        Reopened Reopened Patch Available Patch Available
        7m 38s 1 Carl Steinbach 22/Jul/10 20:34
        Patch Available Patch Available Resolved Resolved
        100d 10h 41m 2 He Yongqiang 27/Jul/10 04:09
        Resolved Resolved Closed Closed
        507d 20h 54m 1 Carl Steinbach 17/Dec/11 00:03
        Carl Steinbach made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Carl Steinbach made changes -
        Affects Version/s 0.6.0 [ 12314524 ]
        Fix Version/s 0.7.0 [ 12315150 ]
        Hide
        He Yongqiang added a comment -
        Show
        He Yongqiang added a comment - typo: HIVE-1452 -> HIVE-1455
        He Yongqiang made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        He Yongqiang added a comment -

        Committed to branch 0.6 together with HIVE-1452. Thanks Paul and Carl!

        Show
        He Yongqiang added a comment - Committed to branch 0.6 together with HIVE-1452 . Thanks Paul and Carl!
        Carl Steinbach made changes -
        Status Reopened [ 4 ] Patch Available [ 10002 ]
        Carl Steinbach made changes -
        Component/s Query Processor [ 12312586 ]
        Carl Steinbach made changes -
        Attachment HIVE-1056-0.6.patch.txt [ 12450202 ]
        Hide
        Carl Steinbach added a comment -

        Attaching patch for branch-0.6

        Show
        Carl Steinbach added a comment - Attaching patch for branch-0.6
        Carl Steinbach made changes -
        Fix Version/s 0.6.0 [ 12314524 ]
        Carl Steinbach made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Carl Steinbach added a comment -

        We should backport this to 0.6

        Show
        Carl Steinbach added a comment - We should backport this to 0.6
        He Yongqiang made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Hide
        He Yongqiang added a comment -

        Committed! Thanks Paul!

        Show
        He Yongqiang added a comment - Committed! Thanks Paul!
        Hide
        He Yongqiang added a comment -

        running tests now.

        Show
        He Yongqiang added a comment - running tests now.
        Paul Yang made changes -
        Attachment HIVE-1056.4.patch [ 12449408 ]
        Hide
        Paul Yang added a comment -

        Added comment for LVF PPD rule.

        Show
        Paul Yang added a comment - Added comment for LVF PPD rule.
        Paul Yang made changes -
        Attachment HIVE-1056.3.patch [ 12449406 ]
        Hide
        Paul Yang added a comment -

        Regenerated again.

        Show
        Paul Yang added a comment - Regenerated again.
        Paul Yang made changes -
        Attachment HIVE-1056.2.patch [ 12449314 ]
        Hide
        Paul Yang added a comment -

        Regenerated patch.

        Show
        Paul Yang added a comment - Regenerated patch.
        Carl Steinbach made changes -
        Fix Version/s 0.7.0 [ 12315150 ]
        Fix Version/s 0.6.0 [ 12314524 ]
        Hide
        HBase Review Board added a comment -

        Message from: "Carl Steinbach" <carl@cloudera.com>

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        http://review.hbase.org/r/198/
        -----------------------------------------------------------

        Review request for Hive Developers.

        Summary
        -------

        Review for patch https://issues.apache.org/jira/secure/attachment/12441230/HIVE-1056.1.patch

        This addresses bug HIVE-1056.
        http://issues.apache.org/jira/browse/HIVE-1056

        Diffs


        contrib/src/java/org/apache/hadoop/hive/contrib/udtf/example/GenericUDTFCount2.java PRE-CREATION
        contrib/src/test/queries/clientpositive/udtf_output_on_close.q PRE-CREATION
        contrib/src/test/results/clientpositive/udtf_output_on_close.q.out PRE-CREATION
        ql/if/queryplan.thrift d387e8e
        ql/src/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java 99d1f2d
        ql/src/gen-php/queryplan_types.php 334b4f8
        ql/src/gen-py/queryplan/ttypes.py d228e68
        ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewForwardOperator.java PRE-CREATION
        ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewJoinOperator.java 371a7ac
        ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 03bd0bb
        ql/src/java/org/apache/hadoop/hive/ql/exec/UDTFOperator.java cab3057
        ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java a2f9dba
        ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewForwardDesc.java PRE-CREATION
        ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 6e1dfe4
        ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicatePushDown.java 19fe5f4
        ql/src/test/queries/clientpositive/lateral_view_ppd.q PRE-CREATION
        ql/src/test/results/clientpositive/lateral_view.q.out 8931455
        ql/src/test/results/clientpositive/lateral_view_ppd.q.out PRE-CREATION

        Diff: http://review.hbase.org/r/198/diff

        Testing
        -------

        Thanks,

        Carl

        Show
        HBase Review Board added a comment - Message from: "Carl Steinbach" <carl@cloudera.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/198/ ----------------------------------------------------------- Review request for Hive Developers. Summary ------- Review for patch https://issues.apache.org/jira/secure/attachment/12441230/HIVE-1056.1.patch This addresses bug HIVE-1056 . http://issues.apache.org/jira/browse/HIVE-1056 Diffs contrib/src/java/org/apache/hadoop/hive/contrib/udtf/example/GenericUDTFCount2.java PRE-CREATION contrib/src/test/queries/clientpositive/udtf_output_on_close.q PRE-CREATION contrib/src/test/results/clientpositive/udtf_output_on_close.q.out PRE-CREATION ql/if/queryplan.thrift d387e8e ql/src/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java 99d1f2d ql/src/gen-php/queryplan_types.php 334b4f8 ql/src/gen-py/queryplan/ttypes.py d228e68 ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewForwardOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewJoinOperator.java 371a7ac ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 03bd0bb ql/src/java/org/apache/hadoop/hive/ql/exec/UDTFOperator.java cab3057 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java a2f9dba ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewForwardDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 6e1dfe4 ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicatePushDown.java 19fe5f4 ql/src/test/queries/clientpositive/lateral_view_ppd.q PRE-CREATION ql/src/test/results/clientpositive/lateral_view.q.out 8931455 ql/src/test/results/clientpositive/lateral_view_ppd.q.out PRE-CREATION Diff: http://review.hbase.org/r/198/diff Testing ------- Thanks, Carl
        John Sichi made changes -
        Fix Version/s 0.6.0 [ 12314524 ]
        John Sichi made changes -
        Fix Version/s 0.6.0 [ 12314524 ]
        Hide
        Ning Zhang added a comment -

        Will take a look.

        Show
        Ning Zhang added a comment - Will take a look.
        Paul Yang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Fix Version/s 0.6.0 [ 12314524 ]
        Paul Yang made changes -
        Field Original Value New Value
        Attachment HIVE-1056.1.patch [ 12441230 ]
        Hide
        Paul Yang added a comment -
        • Enables UDTF's to emit rows on close. Although this was disabled initially, it should be allowed. Without this, the test with lateral views on partitioned tables will fail.
        • Enables PPD for LV. To make pruning easier, a new operator (LateralViewForward) is placed at the head of every LV DAG. Otherwise, it is difficult for the pruner to handle the special case for LV's.
        Show
        Paul Yang added a comment - Enables UDTF's to emit rows on close. Although this was disabled initially, it should be allowed. Without this, the test with lateral views on partitioned tables will fail. Enables PPD for LV. To make pruning easier, a new operator (LateralViewForward) is placed at the head of every LV DAG. Otherwise, it is difficult for the pruner to handle the special case for LV's.
        Paul Yang created issue -

          People

          • Assignee:
            Paul Yang
            Reporter:
            Paul Yang
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development