ashutoshc requested code review of "HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string".
Reviewers: JIRA

https://issues.apache.org/jira/browse/HIVE-2771

This patch adds support for key range scans pushdown to hbase for keys of type string. With this patch filter pushdowns of following types are supported:
a) Point lookups for keys of any types.
b) Range scans for keys of type string.

Test Plan:
Added hbase_ppd_key_range.q which is modeled after hbase_pushdown.q

This is a subtask of HIVE-1643

TEST PLAN
EMPTY

REVISION DETAIL
https://reviews.facebook.net/D1551

AFFECTED FILES
hbase-handler/src/test/results/hbase_ppd_key_range.q.out
hbase-handler/src/test/queries/hbase_ppd_key_range.q
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java

MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/3261/

Tip: use the X-Herald-Rules header to filter Herald messages in your client.

BUILD SUCCESSFUL
Total time: 269 minutes 29 seconds

All the tests passes with this patch. This is ready for review.

jsichi has commented on the revision "HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string".

One thing to note is that if the filter implementation turns out to be too loose (rather than too strict), tests may not notice it because in some cases Hive currently redundantly applies the filter row-by-row on whatever comes back from the scan. (I've added a line-specific comment above for one of those cases.) So you might want to add some private logging just to make sure you've got it right. I can't remember whether that's why I added the WhileMatchFilter originally.

Also, note the comment in HBaseStorageHandler.decomposePredicate. Have you a tested a case such as WHERE key > 20 and key < 30?

INLINE COMMENTS
hbase-handler/src/test/results/hbase_ppd_key_range.q.out:47 Note that here Hive did not remove the redundant filter.

REVISION DETAIL
https://reviews.facebook.net/D1551

ashutoshc has commented on the revision "HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string".

> Also, note the comment in HBaseStorageHandler.decomposePredicate. Have you a tested a case such as WHERE key > 20 and key < 30?

For those queries filter will not be pushed exactly because of that comment. I will update the patch with this test where filter is not pushed in HBase and filtering is done by Hive.

INLINE COMMENTS
hbase-handler/src/test/results/hbase_ppd_key_range.q.out:47 Hive did remove the redundant filter. I verified this by adding private logging. Further, even for queries in hbase_pushdown.q if you look at hbase_pushdown.q.out its the same behavior where for filter key = 90, its been pushed up in storage handler still shows up in explain plan.

REVISION DETAIL
https://reviews.facebook.net/D1551

Isn't a range predicate such as "WHERE key > 20 and key < 30" a fairly important use case?

It is. I plan to take that up in a follow-up jira to keep patch sizes small and manageable. Also, I think HIVE-1634 might have impact on that, so I think it make sense to defer that case.

ashutoshc updated the revision "HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string".
Reviewers: JIRA, jsichi

Added test for key < '20' and key > '30'.

REVISION DETAIL
https://reviews.facebook.net/D1551

AFFECTED FILES
hbase-handler/src/test/results/hbase_ppd_key_range.q.out
hbase-handler/src/test/queries/hbase_ppd_key_range.q
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java

cwsteinbach has accepted the revision "HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string".

+1. Will commit if tests pass.

REVISION DETAIL
https://reviews.facebook.net/D1551

BRANCH
svn

Committed to trunk. Thanks Ashutosh!

ashutoshc has committed the revision "HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string".

Change committed by cws.

REVISION DETAIL
https://reviews.facebook.net/D1551

COMMIT
https://reviews.facebook.net/rHIVE1297675

Patch with license.

This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for
keys of type string
(Ashutosh Chauhan via Carl Steinbach)

Summary:
https://issues.apache.org/jira/browse/HIVE-2771

This patch adds support for key range scans pushdown to hbase for keys of type
string. With this patch filter pushdowns of following types are supported:
a) Point lookups for keys of any types.
b) Range scans for keys of type string.

Test Plan:
Added hbase_ppd_key_range.q which is modeled after hbase_pushdown.q

This is a subtask of HIVE-1643

Test Plan: EMPTY

Reviewers: JIRA, jsichi, cwsteinbach

Reviewed By: cwsteinbach

CC: jsichi, ashutoshc

Differential Revision: https://reviews.facebook.net/D1551 (Revision 1297675)

Result = ABORTED
cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297675
Files :

• /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
• /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
• /hive/trunk/hbase-handler/src/test/queries/hbase_ppd_key_range.q
• /hive/trunk/hbase-handler/src/test/results/hbase_ppd_key_range.q.out
• /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
• /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
• /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java