Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11327

HiveQL to HBase - Predicate Pushdown for composite key not working

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: 0.14.0
    • Fix Version/s: None
    • Component/s: HBase Handler, Hive
    • Labels:
      None
    • Tags:
      hive, predicatepushdown, hbase

      Description

      I am using Hive 0.14 and Hbase 0.98.8 I would like to use HiveQL for accessing a HBase "table".

      I created a table with a complex composite rowkey:


      CREATE EXTERNAL TABLE db.hive_hbase (rowkey struct<p1:string, p2:string, p3:string>, column1 string, column2 string)
      ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
      COLLECTION ITEMS TERMINATED BY ';'
      STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
      WITH SERDEPROPERTIES ("hbase.columns.mapping" =
      ":key,cf:c1,cf:c2")
      TBLPROPERTIES("hbase.table.name"="hbase_table");


      The table is getting successfully created, but the HiveQL query is taking forever:


      SELECT * from db.hive_hbase WHERE rowkey.p1 = 'xyz';


      I am working with 1 TB of data (around 1,5 bn records) and this queries takes forever (It ran over night, but did not finish in the morning).

      I changed the log4j properties to 'DEBUG' and found some interesting information:


      2015-07-15 15:56:41,232 INFO ppd.OpProcFactory (OpProcFactory.java:logExpr(823)) - Pushdown Predicates of FIL For Alias : hive_hbase
      2015-07-15 15:56:41,232 INFO ppd.OpProcFactory (OpProcFactory.java:logExpr(826)) - (rowkey.p1 = 'xyz')


      But some lines later:


      2015-07-15 15:56:41,430 DEBUG ppd.OpProcFactory (OpProcFactory.java:pushFilterToStorageHandler(1051)) - No pushdown possible for predicate: (rowkey.p1 = 'xyz')


      So my guess is: HiveQL over HBase does not do any predicate pushdown but starts a MapReduce job.

      The normal HBase scan (via the HBase Shell) takes around 5 seconds.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              yzuehlke Yannik Zuehlke
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: