[HIVE-1056] Predicate push down does not work with UDTF's - ASF JIRA

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 0.5.0
Fix Version/s: 0.6.0
Component/s: Query Processor
Labels:
None

Hadoop Flags:

Reviewed

Description

Predicate push down does not work with UDTF's in lateral views


hive> SELECT * FROM src LATERAL VIEW explode(array(1,2,3)) myTable AS k WHERE k=1;
FAILED: Unknown exception: null
hive>

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending

Attachments

HIVE-1056.1.patch

08/Apr/10 23:23

78 kB

Paul Yang
HIVE-1056.2.patch

13/Jul/10 02:38

81 kB

Paul Yang
HIVE-1056.3.patch

14/Jul/10 00:18

81 kB

Paul Yang
HIVE-1056.4.patch

14/Jul/10 00:36

82 kB

Paul Yang
HIVE-1056-0.6.patch.txt

22/Jul/10 19:27

83 kB

Carl Steinbach

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Paul Yang added a comment - 08/Apr/10 23:23

Enables UDTF's to emit rows on close. Although this was disabled initially, it should be allowed. Without this, the test with lateral views on partitioned tables will fail.
Enables PPD for LV. To make pruning easier, a new operator (LateralViewForward) is placed at the head of every LV DAG. Otherwise, it is difficult for the pruner to handle the special case for LV's.

Show

Paul Yang added a comment - 08/Apr/10 23:23 Enables UDTF's to emit rows on close. Although this was disabled initially, it should be allowed. Without this, the test with lateral views on partitioned tables will fail. Enables PPD for LV. To make pruning easier, a new operator (LateralViewForward) is placed at the head of every LV DAG. Otherwise, it is difficult for the pruner to handle the special case for LV's.

Hide

Permalink

Ning Zhang added a comment - 13/Apr/10 21:20

Will take a look.

Show

Ning Zhang added a comment - 13/Apr/10 21:20 Will take a look.

Hide

Permalink

HBase Review Board added a comment - 17/Jun/10 21:16

Message from: "Carl Steinbach" <carl@cloudera.com>

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/198/
-----------------------------------------------------------

Review request for Hive Developers.

Summary
-------

Review for patch https://issues.apache.org/jira/secure/attachment/12441230/HIVE-1056.1.patch

This addresses bug ~~HIVE-1056~~.
http://issues.apache.org/jira/browse/HIVE-1056

Diffs

contrib/src/java/org/apache/hadoop/hive/contrib/udtf/example/GenericUDTFCount2.java PRE-CREATION
contrib/src/test/queries/clientpositive/udtf_output_on_close.q PRE-CREATION
contrib/src/test/results/clientpositive/udtf_output_on_close.q.out PRE-CREATION
ql/if/queryplan.thrift d387e8e
ql/src/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java 99d1f2d
ql/src/gen-php/queryplan_types.php 334b4f8
ql/src/gen-py/queryplan/ttypes.py d228e68
ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewForwardOperator.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewJoinOperator.java 371a7ac
ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 03bd0bb
ql/src/java/org/apache/hadoop/hive/ql/exec/UDTFOperator.java cab3057
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java a2f9dba
ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewForwardDesc.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 6e1dfe4
ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicatePushDown.java 19fe5f4
ql/src/test/queries/clientpositive/lateral_view_ppd.q PRE-CREATION
ql/src/test/results/clientpositive/lateral_view.q.out 8931455
ql/src/test/results/clientpositive/lateral_view_ppd.q.out PRE-CREATION

Diff: http://review.hbase.org/r/198/diff

Testing
-------

Thanks,

Carl

Show

HBase Review Board added a comment - 17/Jun/10 21:16 Message from: "Carl Steinbach" <carl@cloudera.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/198/ ----------------------------------------------------------- Review request for Hive Developers. Summary ------- Review for patch https://issues.apache.org/jira/secure/attachment/12441230/HIVE-1056.1.patch This addresses bug HIVE-1056 . http://issues.apache.org/jira/browse/HIVE-1056 Diffs contrib/src/java/org/apache/hadoop/hive/contrib/udtf/example/GenericUDTFCount2.java PRE-CREATION contrib/src/test/queries/clientpositive/udtf_output_on_close.q PRE-CREATION contrib/src/test/results/clientpositive/udtf_output_on_close.q.out PRE-CREATION ql/if/queryplan.thrift d387e8e ql/src/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java 99d1f2d ql/src/gen-php/queryplan_types.php 334b4f8 ql/src/gen-py/queryplan/ttypes.py d228e68 ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewForwardOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewJoinOperator.java 371a7ac ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 03bd0bb ql/src/java/org/apache/hadoop/hive/ql/exec/UDTFOperator.java cab3057 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java a2f9dba ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewForwardDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 6e1dfe4 ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicatePushDown.java 19fe5f4 ql/src/test/queries/clientpositive/lateral_view_ppd.q PRE-CREATION ql/src/test/results/clientpositive/lateral_view.q.out 8931455 ql/src/test/results/clientpositive/lateral_view_ppd.q.out PRE-CREATION Diff: http://review.hbase.org/r/198/diff Testing ------- Thanks, Carl

Hide

Permalink

Paul Yang added a comment - 13/Jul/10 02:38

Regenerated patch.

Show

Paul Yang added a comment - 13/Jul/10 02:38 Regenerated patch.

Hide

Permalink

Paul Yang added a comment - 14/Jul/10 00:18

Regenerated again.

Show

Paul Yang added a comment - 14/Jul/10 00:18 Regenerated again.

Hide

Permalink

Paul Yang added a comment - 14/Jul/10 00:36

Added comment for LVF PPD rule.

Show

Paul Yang added a comment - 14/Jul/10 00:36 Added comment for LVF PPD rule.

Hide

Permalink

He Yongqiang added a comment - 14/Jul/10 00:54

running tests now.

Show

He Yongqiang added a comment - 14/Jul/10 00:54 running tests now.

Hide

Permalink

He Yongqiang added a comment - 14/Jul/10 02:30

Committed! Thanks Paul!

Show

He Yongqiang added a comment - 14/Jul/10 02:30 Committed! Thanks Paul!

Hide

Permalink

Carl Steinbach added a comment - 22/Jul/10 19:26

We should backport this to 0.6

Show

Carl Steinbach added a comment - 22/Jul/10 19:26 We should backport this to 0.6

Hide

Permalink

Carl Steinbach added a comment - 22/Jul/10 19:27

Attaching patch for branch-0.6

Show

Carl Steinbach added a comment - 22/Jul/10 19:27 Attaching patch for branch-0.6

Hide

Permalink

He Yongqiang added a comment - 27/Jul/10 03:09

Committed to branch 0.6 together with ~~HIVE-1452~~. Thanks Paul and Carl!

Show

He Yongqiang added a comment - 27/Jul/10 03:09 Committed to branch 0.6 together with HIVE-1452 . Thanks Paul and Carl!

Hide

Permalink

He Yongqiang added a comment - 27/Jul/10 03:10

typo:
~~HIVE-1452~~ ->~~HIVE-1455~~

Show

He Yongqiang added a comment - 27/Jul/10 03:10 typo: HIVE-1452 -> HIVE-1455

People

Assignee:

Paul Yang

Reporter:

Paul Yang

Votes:

1 Vote for this issue

Watchers:

2 Start watching this issue

Dates

Created:

16/Jan/10 01:32

Updated:

17/Dec/11 00:03

Resolved:

27/Jul/10 03:09

Development

Agile

View on Board