Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-19850

Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error

Log workAgile BoardRank to TopRank to BottomVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 4.0.0
    • Component/s: Tez
    • Labels:
      None

      Description

       

      When multiple views are used along with union all, it is resulting in the following error when dynamic partition pruning is enabled in tez. 

       

      Exception in thread "main" java.lang.AssertionError: No work found for tablescan TS[8]
       at org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
       at org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
       at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
       at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
       at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
       at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
       at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
       at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
       at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
       at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
       at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
       at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1130)
       at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
       at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:204)
       at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:433)
       at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:894)
       at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:825)
       at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:726)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:606)
       at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
       at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

       

      Steps to reproduce:

      set hive.execution.engine=tez;

      set hive.tez.dynamic.partition.pruning=true;

      CREATE TABLE t1(key string, value string, c_int int, c_float float, c_boolean boolean) partitioned by (dt string);

      CREATE TABLE t2(key string, value string, c_int int, c_float float, c_boolean boolean) partitioned by (dt string);

      CREATE TABLE t3(key string, value string, c_int int, c_float float, c_boolean boolean) partitioned by (dt string);

       

      insert into table t1 partition(dt='2018') values ('k1','v1',1,1.0,true);

      insert into table t2 partition(dt='2018') values ('k2','v2',2,2.0,true);

      insert into table t3 partition(dt='2018') values ('k3','v3',3,3.0,true);

       

      CREATE VIEW `view1` AS select `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt` from `t1` union all select `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt` from `t2`;

      CREATE VIEW `view2` AS select `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt` from `t2` union all select `t3`.`key`,`t3`.`value`,`t3`.`c_int`,`t3`.`c_float`,`t3`.`c_boolean`,`t3`.`dt` from `t3`;

      create table t4 as select key,value,c_int,c_float,c_boolean,dt from t1 union all select v1.key,v1.value,v1.c_int,v1.c_float,v1.c_boolean,v1.dt from view1 v1 join view2 v2 on v1.dt=v2.dt;

      CREATE VIEW `view3` AS select `t4`.`key`,`t4`.`value`,`t4`.`c_int`,`t4`.`c_float`,`t4`.`c_boolean`,`t4`.`dt` from `t4` union all select `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt` from `t1`;

       

      select count(0) from view2 v2 join view3 v3 on v2.dt=v3.dt; // Throws No work found for tablescan error

        Attachments

        1. HIVE-19850.patch
          1 kB
          Ganesha Shreedhara

        Issue Links

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              ganeshas Ganesha Shreedhara Assign to me
              Reporter:
              ganeshas Ganesha Shreedhara

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment