Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0, 0.7.1, 0.8.0, 0.8.1
    • Fix Version/s: 0.9.0
    • Component/s: HBase Handler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This is a subtask of HIVE-1643

      1. hive-2771.patch
        31 kB
        Ashutosh Chauhan
      2. ASF.LICENSE.NOT.GRANTED--HIVE-2771.D1551.2.patch
        31 kB
        Phabricator
      3. ASF.LICENSE.NOT.GRANTED--HIVE-2771.D1551.1.patch
        29 kB
        Phabricator

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
          HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for
          keys of type string
          (Ashutosh Chauhan via Carl Steinbach)

          Summary:
          https://issues.apache.org/jira/browse/HIVE-2771

          This patch adds support for key range scans pushdown to hbase for keys of type
          string. With this patch filter pushdowns of following types are supported:
          a) Point lookups for keys of any types.
          b) Range scans for keys of type string.

          Test Plan:
          Added hbase_ppd_key_range.q which is modeled after hbase_pushdown.q

          This is a subtask of HIVE-1643

          Test Plan: EMPTY

          Reviewers: JIRA, jsichi, cwsteinbach

          Reviewed By: cwsteinbach

          CC: jsichi, ashutoshc

          Differential Revision: https://reviews.facebook.net/D1551 (Revision 1297675)

          Result = ABORTED
          cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297675
          Files :

          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
          • /hive/trunk/hbase-handler/src/test/queries/hbase_ppd_key_range.q
          • /hive/trunk/hbase-handler/src/test/results/hbase_ppd_key_range.q.out
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string (Ashutosh Chauhan via Carl Steinbach) Summary: https://issues.apache.org/jira/browse/HIVE-2771 This patch adds support for key range scans pushdown to hbase for keys of type string. With this patch filter pushdowns of following types are supported: a) Point lookups for keys of any types. b) Range scans for keys of type string. Test Plan: Added hbase_ppd_key_range.q which is modeled after hbase_pushdown.q This is a subtask of HIVE-1643 Test Plan: EMPTY Reviewers: JIRA, jsichi, cwsteinbach Reviewed By: cwsteinbach CC: jsichi, ashutoshc Differential Revision: https://reviews.facebook.net/D1551 (Revision 1297675) Result = ABORTED cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297675 Files : /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java /hive/trunk/hbase-handler/src/test/queries/hbase_ppd_key_range.q /hive/trunk/hbase-handler/src/test/results/hbase_ppd_key_range.q.out /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
          Ashutosh Chauhan made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Ashutosh Chauhan added a comment -

          This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

          Show
          Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
          Ashutosh Chauhan made changes -
          Attachment hive-2771.patch [ 12517367 ]
          Hide
          Ashutosh Chauhan added a comment -

          Patch with license.

          Show
          Ashutosh Chauhan added a comment - Patch with license.
          Hide
          Phabricator added a comment -

          ashutoshc has committed the revision "HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string".

          Change committed by cws.

          REVISION DETAIL
          https://reviews.facebook.net/D1551

          COMMIT
          https://reviews.facebook.net/rHIVE1297675

          Show
          Phabricator added a comment - ashutoshc has committed the revision " HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string". Change committed by cws. REVISION DETAIL https://reviews.facebook.net/D1551 COMMIT https://reviews.facebook.net/rHIVE1297675
          Carl Steinbach made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.9.0 [ 12317742 ]
          Resolution Fixed [ 1 ]
          Hide
          Carl Steinbach added a comment -

          Committed to trunk. Thanks Ashutosh!

          Show
          Carl Steinbach added a comment - Committed to trunk. Thanks Ashutosh!
          Hide
          Phabricator added a comment -

          cwsteinbach has accepted the revision "HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string".

          +1. Will commit if tests pass.

          REVISION DETAIL
          https://reviews.facebook.net/D1551

          BRANCH
          svn

          Show
          Phabricator added a comment - cwsteinbach has accepted the revision " HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string". +1. Will commit if tests pass. REVISION DETAIL https://reviews.facebook.net/D1551 BRANCH svn
          Ashutosh Chauhan made changes -
          Link This issue blocks HIVE-2815 [ HIVE-2815 ]
          Phabricator made changes -
          Attachment HIVE-2771.D1551.2.patch [ 12513842 ]
          Hide
          Phabricator added a comment -

          ashutoshc updated the revision "HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string".
          Reviewers: JIRA, jsichi

          Added test for key < '20' and key > '30'.

          REVISION DETAIL
          https://reviews.facebook.net/D1551

          AFFECTED FILES
          hbase-handler/src/test/results/hbase_ppd_key_range.q.out
          hbase-handler/src/test/queries/hbase_ppd_key_range.q
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
          ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java

          Show
          Phabricator added a comment - ashutoshc updated the revision " HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string". Reviewers: JIRA, jsichi Added test for key < '20' and key > '30'. REVISION DETAIL https://reviews.facebook.net/D1551 AFFECTED FILES hbase-handler/src/test/results/hbase_ppd_key_range.q.out hbase-handler/src/test/queries/hbase_ppd_key_range.q hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
          Hide
          Ashutosh Chauhan added a comment -

          It is. I plan to take that up in a follow-up jira to keep patch sizes small and manageable. Also, I think HIVE-1634 might have impact on that, so I think it make sense to defer that case.

          Show
          Ashutosh Chauhan added a comment - It is. I plan to take that up in a follow-up jira to keep patch sizes small and manageable. Also, I think HIVE-1634 might have impact on that, so I think it make sense to defer that case.
          Hide
          John Sichi added a comment -

          Isn't a range predicate such as "WHERE key > 20 and key < 30" a fairly important use case?

          Show
          John Sichi added a comment - Isn't a range predicate such as "WHERE key > 20 and key < 30" a fairly important use case?
          Hide
          Phabricator added a comment -

          ashutoshc has commented on the revision "HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string".

          > Also, note the comment in HBaseStorageHandler.decomposePredicate. Have you a tested a case such as WHERE key > 20 and key < 30?

          For those queries filter will not be pushed exactly because of that comment. I will update the patch with this test where filter is not pushed in HBase and filtering is done by Hive.

          INLINE COMMENTS
          hbase-handler/src/test/results/hbase_ppd_key_range.q.out:47 Hive did remove the redundant filter. I verified this by adding private logging. Further, even for queries in hbase_pushdown.q if you look at hbase_pushdown.q.out its the same behavior where for filter key = 90, its been pushed up in storage handler still shows up in explain plan.

          REVISION DETAIL
          https://reviews.facebook.net/D1551

          Show
          Phabricator added a comment - ashutoshc has commented on the revision " HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string". > Also, note the comment in HBaseStorageHandler.decomposePredicate. Have you a tested a case such as WHERE key > 20 and key < 30? For those queries filter will not be pushed exactly because of that comment. I will update the patch with this test where filter is not pushed in HBase and filtering is done by Hive. INLINE COMMENTS hbase-handler/src/test/results/hbase_ppd_key_range.q.out:47 Hive did remove the redundant filter. I verified this by adding private logging. Further, even for queries in hbase_pushdown.q if you look at hbase_pushdown.q.out its the same behavior where for filter key = 90, its been pushed up in storage handler still shows up in explain plan. REVISION DETAIL https://reviews.facebook.net/D1551
          Hide
          Phabricator added a comment -

          jsichi has commented on the revision "HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string".

          One thing to note is that if the filter implementation turns out to be too loose (rather than too strict), tests may not notice it because in some cases Hive currently redundantly applies the filter row-by-row on whatever comes back from the scan. (I've added a line-specific comment above for one of those cases.) So you might want to add some private logging just to make sure you've got it right. I can't remember whether that's why I added the WhileMatchFilter originally.

          Also, note the comment in HBaseStorageHandler.decomposePredicate. Have you a tested a case such as WHERE key > 20 and key < 30?

          INLINE COMMENTS
          hbase-handler/src/test/results/hbase_ppd_key_range.q.out:47 Note that here Hive did not remove the redundant filter.

          REVISION DETAIL
          https://reviews.facebook.net/D1551

          Show
          Phabricator added a comment - jsichi has commented on the revision " HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string". One thing to note is that if the filter implementation turns out to be too loose (rather than too strict), tests may not notice it because in some cases Hive currently redundantly applies the filter row-by-row on whatever comes back from the scan. (I've added a line-specific comment above for one of those cases.) So you might want to add some private logging just to make sure you've got it right. I can't remember whether that's why I added the WhileMatchFilter originally. Also, note the comment in HBaseStorageHandler.decomposePredicate. Have you a tested a case such as WHERE key > 20 and key < 30? INLINE COMMENTS hbase-handler/src/test/results/hbase_ppd_key_range.q.out:47 Note that here Hive did not remove the redundant filter. REVISION DETAIL https://reviews.facebook.net/D1551
          Ashutosh Chauhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Ashutosh Chauhan added a comment -
          BUILD SUCCESSFUL
          Total time: 269 minutes 29 seconds
          

          All the tests passes with this patch. This is ready for review.

          Show
          Ashutosh Chauhan added a comment - BUILD SUCCESSFUL Total time: 269 minutes 29 seconds All the tests passes with this patch. This is ready for review.
          Phabricator made changes -
          Attachment HIVE-2771.D1551.1.patch [ 12512716 ]
          Hide
          Phabricator added a comment -

          ashutoshc requested code review of "HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string".
          Reviewers: JIRA

          https://issues.apache.org/jira/browse/HIVE-2771

          This patch adds support for key range scans pushdown to hbase for keys of type string. With this patch filter pushdowns of following types are supported:
          a) Point lookups for keys of any types.
          b) Range scans for keys of type string.

          Test Plan:
          Added hbase_ppd_key_range.q which is modeled after hbase_pushdown.q

          This is a subtask of HIVE-1643

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D1551

          AFFECTED FILES
          hbase-handler/src/test/results/hbase_ppd_key_range.q.out
          hbase-handler/src/test/queries/hbase_ppd_key_range.q
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
          ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/3261/

          Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          Show
          Phabricator added a comment - ashutoshc requested code review of " HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2771 This patch adds support for key range scans pushdown to hbase for keys of type string. With this patch filter pushdowns of following types are supported: a) Point lookups for keys of any types. b) Range scans for keys of type string. Test Plan: Added hbase_ppd_key_range.q which is modeled after hbase_pushdown.q This is a subtask of HIVE-1643 TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1551 AFFECTED FILES hbase-handler/src/test/results/hbase_ppd_key_range.q.out hbase-handler/src/test/queries/hbase_ppd_key_range.q hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/3261/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
          Ashutosh Chauhan made changes -
          Field Original Value New Value
          Parent HIVE-1643 [ 12474221 ]
          Issue Type New Feature [ 2 ] Sub-task [ 7 ]
          Ashutosh Chauhan created issue -

            People

            • Assignee:
              Ashutosh Chauhan
              Reporter:
              Ashutosh Chauhan
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development