Hive
  1. Hive
  2. HIVE-1226

support filter pushdown against non-native tables

    Details

      Description

      For example, HBase's scan object can take filters.

      1. HIVE-1226.1.patch
        27 kB
        John Sichi
      2. HIVE-1226.2.patch
        64 kB
        John Sichi
      3. HIVE-1226.3.patch
        65 kB
        John Sichi
      4. HIVE-1226.4.patch
        65 kB
        John Sichi

        Issue Links

          Activity

          Hide
          John Sichi added a comment -
          Show
          John Sichi added a comment - Design doc draft is here: http://wiki.apache.org/hadoop/Hive/FilterPushdownDev
          Hide
          John Sichi added a comment -

          This patch is nowhere near done; just checkpointing what I've been working on so far.

          Show
          John Sichi added a comment - This patch is nowhere near done; just checkpointing what I've been working on so far.
          Hide
          John Sichi added a comment -

          Almost there...but just realized that I haven't included the predicate analysis in getSplits, and we need it there (e.g. in the HBase case to avoid generating splits for regions which can't possibly contain the key). So one more patch coming after this and then it's ready for review.

          Show
          John Sichi added a comment - Almost there...but just realized that I haven't included the predicate analysis in getSplits, and we need it there (e.g. in the HBase case to avoid generating splits for regions which can't possibly contain the key). So one more patch coming after this and then it's ready for review.
          Hide
          John Sichi added a comment -

          This patch deals with passing the filters down to getSplits too.

          Show
          John Sichi added a comment - This patch deals with passing the filters down to getSplits too.
          Hide
          John Sichi added a comment -

          I'm still running through tests and updating the doc, but the patch is available for review:

          https://review.cloudera.org/r/823/

          Show
          John Sichi added a comment - I'm still running through tests and updating the doc, but the patch is available for review: https://review.cloudera.org/r/823/
          Hide
          HBase Review Board added a comment -

          Message from: "John Sichi" <jsichi@facebook.com>

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.cloudera.org/r/823/
          -----------------------------------------------------------

          Review request for Hive Developers.

          Summary
          -------

          review request from JVS

          This addresses bug HIVE-1226.
          http://issues.apache.org/jira/browse/HIVE-1226

          Diffs


          http://svn.apache.org/repos/asf/hadoop/hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 996612
          http://svn.apache.org/repos/asf/hadoop/hive/trunk/conf/hive-default.xml 996612
          http://svn.apache.org/repos/asf/hadoop/hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 996612
          http://svn.apache.org/repos/asf/hadoop/hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 996612
          http://svn.apache.org/repos/asf/hadoop/hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 996612
          http://svn.apache.org/repos/asf/hadoop/hive/trunk/hbase-handler/src/test/queries/hbase_pushdown.q PRE-CREATION
          http://svn.apache.org/repos/asf/hadoop/hive/trunk/hbase-handler/src/test/results/hbase_pushdown.q.out PRE-CREATION
          http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 996612
          http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java PRE-CREATION
          http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/index/IndexSearchCondition.java PRE-CREATION
          http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/BucketizedHiveInputFormat.java 996612
          http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 996612
          http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 996612
          http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStoragePredicateHandler.java PRE-CREATION
          http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 996612
          http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 996612
          http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpWalkerInfo.java 996612
          http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicatePushDown.java 996612

          Diff: http://review.cloudera.org/r/823/diff

          Testing
          -------

          Thanks,

          John

          Show
          HBase Review Board added a comment - Message from: "John Sichi" <jsichi@facebook.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.cloudera.org/r/823/ ----------------------------------------------------------- Review request for Hive Developers. Summary ------- review request from JVS This addresses bug HIVE-1226 . http://issues.apache.org/jira/browse/HIVE-1226 Diffs http://svn.apache.org/repos/asf/hadoop/hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 996612 http://svn.apache.org/repos/asf/hadoop/hive/trunk/conf/hive-default.xml 996612 http://svn.apache.org/repos/asf/hadoop/hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 996612 http://svn.apache.org/repos/asf/hadoop/hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 996612 http://svn.apache.org/repos/asf/hadoop/hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 996612 http://svn.apache.org/repos/asf/hadoop/hive/trunk/hbase-handler/src/test/queries/hbase_pushdown.q PRE-CREATION http://svn.apache.org/repos/asf/hadoop/hive/trunk/hbase-handler/src/test/results/hbase_pushdown.q.out PRE-CREATION http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 996612 http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/index/IndexSearchCondition.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/BucketizedHiveInputFormat.java 996612 http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 996612 http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 996612 http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStoragePredicateHandler.java PRE-CREATION http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 996612 http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 996612 http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpWalkerInfo.java 996612 http://svn.apache.org/repos/asf/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicatePushDown.java 996612 Diff: http://review.cloudera.org/r/823/diff Testing ------- Thanks, John
          Hide
          John Sichi added a comment -

          Updated patch with one tiny bugfix found while testing; this version passes all tests.

          HiveInputFormat.java

          + if ((aliases != null) && (aliases.size() == 1)) {

          Show
          John Sichi added a comment - Updated patch with one tiny bugfix found while testing; this version passes all tests. HiveInputFormat.java + if ((aliases != null) && (aliases.size() == 1)) {
          Hide
          He Yongqiang added a comment -

          The patch looks good.

          One question:
          In HBaseStorageHandler, it will exit if searchConditions.size() != 1.
          This makes sense if there are two point predicates on the key column (connected by 'AND'). What if they are composed to perform a range query (like a<key<b) ?

          Can you also open another jira for indexing to leverage this change? Indexing needs this change to do automatic rewrite of user's query.

          Show
          He Yongqiang added a comment - The patch looks good. One question: In HBaseStorageHandler, it will exit if searchConditions.size() != 1. This makes sense if there are two point predicates on the key column (connected by 'AND'). What if they are composed to perform a range query (like a<key<b) ? Can you also open another jira for indexing to leverage this change? Indexing needs this change to do automatic rewrite of user's query.
          Hide
          John Sichi added a comment - - edited

          Regarding "What if they are composed to perform a range query (like a<key<b) ?": for now, we are only supporting exact equality predicates. I am going to open a followup JIRA for range predicates.

          Show
          John Sichi added a comment - - edited Regarding "What if they are composed to perform a range query (like a<key<b) ?": for now, we are only supporting exact equality predicates. I am going to open a followup JIRA for range predicates.
          Hide
          John Sichi added a comment -

          Created followup HIVE-1643 for range scans and non-rowkey predicates.

          Show
          John Sichi added a comment - Created followup HIVE-1643 for range scans and non-rowkey predicates.
          Hide
          John Sichi added a comment -

          Created followup HIVE-1644 for indexing.

          Show
          John Sichi added a comment - Created followup HIVE-1644 for indexing.
          Hide
          He Yongqiang added a comment -

          +1, will commit after tests pass.

          Show
          He Yongqiang added a comment - +1, will commit after tests pass.
          Hide
          He Yongqiang added a comment -

          I just committed! Thanks John!

          Show
          He Yongqiang added a comment - I just committed! Thanks John!

            People

            • Assignee:
              John Sichi
              Reporter:
              John Sichi
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development