[HIVE-19850] Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Tez
• Assignee: Ganesha Shreedhara Assign to me
• Reporter: Ganesha Shreedhara
• Created: 11/Jun/18 05:18
• Updated: 17/Nov/22 08:55
• Resolved: 05/Jul/18 06:04

Description

 

When multiple views are used along with union all, it is resulting in the following error when dynamic partition pruning is enabled in tez. 

 

Exception in thread "main" java.lang.AssertionError: No work found for tablescan TS[8]
 at org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
 at org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
 at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
 at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
 at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
 at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
 at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
 at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
 at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
 at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1130)
 at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
 at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:204)
 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:433)
 at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:894)
 at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:825)
 at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:726)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
 at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

 

Steps to reproduce:

set hive.execution.engine=tez;

set hive.tez.dynamic.partition.pruning=true;

CREATE TABLE t1(key string, value string, c_int int, c_float float, c_boolean boolean) partitioned by (dt string);

CREATE TABLE t2(key string, value string, c_int int, c_float float, c_boolean boolean) partitioned by (dt string);

CREATE TABLE t3(key string, value string, c_int int, c_float float, c_boolean boolean) partitioned by (dt string);

 

insert into table t1 partition(dt='2018') values ('k1','v1',1,1.0,true);

insert into table t2 partition(dt='2018') values ('k2','v2',2,2.0,true);

insert into table t3 partition(dt='2018') values ('k3','v3',3,3.0,true);

 

CREATE VIEW `view1` AS select `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt` from `t1` union all select `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt` from `t2`;

CREATE VIEW `view2` AS select `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt` from `t2` union all select `t3`.`key`,`t3`.`value`,`t3`.`c_int`,`t3`.`c_float`,`t3`.`c_boolean`,`t3`.`dt` from `t3`;

create table t4 as select key,value,c_int,c_float,c_boolean,dt from t1 union all select v1.key,v1.value,v1.c_int,v1.c_float,v1.c_boolean,v1.dt from view1 v1 join view2 v2 on v1.dt=v2.dt;

CREATE VIEW `view3` AS select `t4`.`key`,`t4`.`value`,`t4`.`c_int`,`t4`.`c_float`,`t4`.`c_boolean`,`t4`.`dt` from `t4` union all select `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt` from `t1`;

 

select count(0) from view2 v2 join view3 v3 on v2.dt=v3.dt; // Throws No work found for tablescan error

Issue Links

• relates to: HIVE-15230 query form a view join another view happens "No work found for tablescan TS"