[HIVE-3104] Predicate pushdown doesn't work with multi-insert statements using LATERAL VIEW

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.9.0
• Component/s: CLI

Environment

Apache Hive 0.9.0, Apache Hadoop 0.20.205.0

• Tags: predicate pushdown, multi-insert, lateral view, udtf
• Assignee: Xuefu Zhang
• Reporter: Mark Grover
• Created: 08/Jun/12 04:09
• Updated: 29/Aug/13 21:58

Description

Predicate pushdown seems to work for single-insert queries using LATERAL VIEW. It also seems to work for multi-insert queries not using LATERAL VIEW. However, it doesn't work for multi-insert queries using LATERAL VIEW: It errors out right away with 'FAILED: SemanticException [Error 10041]: No partition predicate found for Alias "test" Table "test"'.

Here are some examples. In the below examples, I make use of the fact that a query with no partition filtering when run under "hive.mapred.mode=strict" fails.

--Table creation and population
DROP TABLE IF EXISTS test;
CREATE TABLE test (col1 array<int>, col2 int) PARTITIONED BY (part_col int);
INSERT OVERWRITE TABLE test PARTITION (part_col=1) SELECT array(1,2), count FROM test;
INSERT OVERWRITE TABLE test PARTITION (part_col=2) SELECT array(2,4,6), count FROM test;

– Query 1
– This succeeds (using LATERAL VIEW with single insert)
set hive.mapred.mode=strict;
FROM test
LATERAL VIEW explode(col1) tmp AS exp_col1
INSERT OVERWRITE DIRECTORY '/test/1'
SELECT exp_col1
WHERE (part_col=2);

– Query 2
– This succeeds (NOT using LATERAL VIEW with multi-insert)
set hive.mapred.mode=strict;
FROM test
INSERT OVERWRITE DIRECTORY '/test/1'
SELECT col1
WHERE (part_col=2)
INSERT OVERWRITE DIRECTORY '/test/2'
SELECT col1
WHERE (part_col=2);

– Query 3
– This fails (using LATERAL VIEW with multi-insert)
set hive.mapred.mode=strict;
FROM test
LATERAL VIEW explode(col1) tmp AS exp_col1
INSERT OVERWRITE DIRECTORY '/test/1'
SELECT exp_col1
WHERE (part_col=2)
INSERT OVERWRITE DIRECTORY '/test/2'
SELECT exp_col1
WHERE (part_col=2);