Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-10545

Implement predicate pushdown for queries over HBase snapshots

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: HBase Handler
    • Labels:
      None

      Description

      Hive's hbase integration currently supports queries over HBase snapshots, and predicate pushdown for queries over HBase tables, but doesn't currently support predicate pushdown for queries over HBase snapshots. This seems to be largely due to the fact that the hbase handler uses the `mapred` TableSnapshotInputFormat implementation, which doesn't support pushing a scan to the job, and not the `mapreduce` implementation, which does (see https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.html#initTableSnapshotMapJob(java.lang.String,%20java.lang.String,%20java.lang.Class,%20java.lang.Class,%20java.lang.Class,%20org.apache.hadoop.mapred.JobConf,%20boolean,%20org.apache.hadoop.fs.Path vs https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.html#initTableSnapshotMapperJob(java.lang.String,%20org.apache.hadoop.hbase.client.Scan,%20java.lang.Class,%20java.lang.Class,%20java.lang.Class,%20org.apache.hadoop.mapreduce.Job,%20boolean,%20org.apache.hadoop.fs.Path)) .

      Hive should be able to switch to the mapreduce implementation (performing the necessary shimming between mapred and mapreduce), and thus gain the ability to push predicates down to the input format in the same way as is done with HiveTableInputFormat. This switch should result in significant performance improvements for queries which specify range/equality conditions on the row key (which seems like it would be a reasonably common case).

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              amains12 Andrew Mains
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: