Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-19850

Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 4.0.0
    • Component/s: Tez
    • Labels:
      None

      Description

       

      When multiple views are used along with union all, it is resulting in the following error when dynamic partition pruning is enabled in tez. 

       

      Exception in thread "main" java.lang.AssertionError: No work found for tablescan TS[8]
       at org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
       at org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
       at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
       at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
       at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
       at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
       at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
       at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
       at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
       at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
       at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
       at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1130)
       at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
       at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:204)
       at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:433)
       at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:894)
       at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:825)
       at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:726)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:606)
       at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
       at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

       

      Steps to reproduce:

      set hive.execution.engine=tez;

      set hive.tez.dynamic.partition.pruning=true;

      CREATE TABLE t1(key string, value string, c_int int, c_float float, c_boolean boolean) partitioned by (dt string);

      CREATE TABLE t2(key string, value string, c_int int, c_float float, c_boolean boolean) partitioned by (dt string);

      CREATE TABLE t3(key string, value string, c_int int, c_float float, c_boolean boolean) partitioned by (dt string);

       

      insert into table t1 partition(dt='2018') values ('k1','v1',1,1.0,true);

      insert into table t2 partition(dt='2018') values ('k2','v2',2,2.0,true);

      insert into table t3 partition(dt='2018') values ('k3','v3',3,3.0,true);

       

      CREATE VIEW `view1` AS select `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt` from `t1` union all select `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt` from `t2`;

      CREATE VIEW `view2` AS select `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt` from `t2` union all select `t3`.`key`,`t3`.`value`,`t3`.`c_int`,`t3`.`c_float`,`t3`.`c_boolean`,`t3`.`dt` from `t3`;

      create table t4 as select key,value,c_int,c_float,c_boolean,dt from t1 union all select v1.key,v1.value,v1.c_int,v1.c_float,v1.c_boolean,v1.dt from view1 v1 join view2 v2 on v1.dt=v2.dt;

      CREATE VIEW `view3` AS select `t4`.`key`,`t4`.`value`,`t4`.`c_int`,`t4`.`c_float`,`t4`.`c_boolean`,`t4`.`dt` from `t4` union all select `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt` from `t1`;

       

      select count(0) from view2 v2 join view3 v3 on v2.dt=v3.dt; // Throws No work found for tablescan error

        Attachments

        1. HIVE-19850.patch
          1 kB
          Ganesha Shreedhara

          Issue Links

            Activity

              People

              • Assignee:
                ganeshas Ganesha Shreedhara
                Reporter:
                ganeshas Ganesha Shreedhara
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: