Details
-
Type:
Improvement
-
Status: Resolved
-
Priority:
Trivial
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 1.1.0
-
Component/s: HBase Handler
-
Labels:
Description
Originated from HIVE-2781 and not accepted, but I think this could be helpful to someone.
By using special column notation ':timestamp' in HBASE_COLUMNS_MAPPING, user might access timestamp value in hbase KeyValue.
CREATE TABLE hbase_table (key int, value string, time timestamp) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf:string,:timestamp")
Issue Links
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Does HBase integration in Hive allows retrieval of multiple versions ( diff time stamp ) of the same cell?
Also whether allows specifying the time range while select such that the cell value versions within this time range will be retrieved?
No and no as far as I know, but might be implemented by modifying HiveHBaseTableInputFormat and HBaseStorageHandler.
navis updated the revision "HIVE-2828 [jira] make timestamp accessible in the hbase KeyValue".
Reviewers: JIRA
Applied HIVE-2771
REVISION DETAIL
https://reviews.facebook.net/D1989
AFFECTED FILES
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java.rej
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java.rej
hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java
hbase-handler/src/test/queries/external_table_ppd.q
hbase-handler/src/test/queries/hbase_timestamp.q
hbase-handler/src/test/results/external_table_ppd.q.out
hbase-handler/src/test/results/hbase_timestamp.q.out
navis updated the revision "HIVE-2828 [jira] make timestamp accessible in the hbase KeyValue".
Reviewers: JIRA
1. Rebased on trunk
2. Support string type for timestamp column
REVISION DETAIL
https://reviews.facebook.net/D1989
AFFECTED FILES
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java
hbase-handler/src/test/queries/hbase_timestamp.q
hbase-handler/src/test/results/hbase_timestamp.q.out
navis updated the revision "HIVE-2828 [jira] make timestamp accessible in the hbase KeyValue".
Reviewers: JIRA
Rebased on trunk
REVISION DETAIL
https://reviews.facebook.net/D1989
AFFECTED FILES
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java
hbase-handler/src/test/queries/hbase_timestamp.q
hbase-handler/src/test/results/hbase_timestamp.q.out
navis updated the revision "HIVE-2828 [jira] make timestamp accessible in the hbase KeyValue".
Reviewers: JIRA
Rebased on trunk
REVISION DETAIL
https://reviews.facebook.net/D1989
AFFECTED FILES
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java
hbase-handler/src/test/queries/hbase_timestamp.q
hbase-handler/src/test/results/hbase_timestamp.q.out
Would this allow aggregation across timestamps as well?
If what you meant was is it possible to 'group by timestamp', it's possible.
What is the right process for applying these patches? Also, on which SVN paths would it be applicable? Obviously the code has changed since these were created so I'm trying to figure out how to apply them. We currently have a version of 0.8.0 that would be great to apply these patches against. I tried against branches/0.8.0 to no avail.
Thanks!
John
What is the status of this ticket?
Can someone tell me if this is going to be pushed? or any reason why its not pushed?
This is a good feature and we have come to a couple of situations where this support is good to have.
The current patch might not be applicable to the current version Hive 0.13, but I can work on it (in-fact I have a patch available with the current trunk but slightly different implementation).
Overall: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12665836/HIVE-2828.6.patch.txt
ERROR: -1 due to 4 failed/errored test(s), 6134 tests executed
Failed tests:
org.apache.hadoop.hive.hbase.TestLazyHBaseObject.testLazyHBaseRow1 org.apache.hadoop.hive.hbase.TestLazyHBaseObject.testLazyHBaseRow2 org.apache.hadoop.hive.hbase.TestLazyHBaseObject.testLazyHBaseRow3 org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation
Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/595/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/595/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-595/
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed
This message is automatically generated.
ATTACHMENT ID: 12665836
Overall: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12665849/HIVE-2828.7.patch.txt
ERROR: -1 due to 1 failed/errored test(s), 6135 tests executed
Failed tests:
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation
Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/598/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/598/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-598/
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed
This message is automatically generated.
ATTACHMENT ID: 12665849
Overall: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12672261/HIVE-2828.8.patch.txt
SUCCESS: +1 6488 tests passed
Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1072/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1072/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1072/
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase
This message is automatically generated.
ATTACHMENT ID: 12672261
Sushanth Sowmyan I think there are some requests on this. Could you review this if time allows?
Sure, I'll try to look into this tonight.
Sorry, I've been swamped the last couple of days, and expect to be so for the next couple of days - this weekend is going to be the earliest I can look at it. I think Brock Noland and Nick Dimiduk also potentially have some context, so tagging them if they want to look at this before I do so.
Swarnim Kulkarni has worked on the hbase serde lately.
Sushanth Sowmyan, Brock Noland, Swarnim Kulkarni, Sergey
Also, HIVE-8267 is the sequel feature request, with the ability to make timestamp of any cell available. So could you guys also look into that?
Sure. I will take a look at it today.
Left comments on the phabricator review.
Left a comment on phabricator. +1
ndimiduk added a subscriber: ndimiduk.
ndimiduk accepted this revision.
ndimiduk added a reviewer: ndimiduk.
ndimiduk added a comment.
This revision is now accepted and ready to land.
+1
BRANCH
DPAL-861
INLINE COMMENTS
hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java:154 This is as good a decision as any. Maybe down the road someone will want to specify which cell's timestamp maps to which column?
REVISION DETAIL
https://reviews.facebook.net/D1989
To: navis, JIRA, ndimiduk
Cc: ndimiduk
Nick Dimiduk It would be handled in HIVE-8267. (see https://issues.apache.org/jira/browse/HIVE-8267?focusedCommentId=14154347&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14154347)
Swarnim Kulkarni Cannot find your comments in phabricator and the patch in it would have gone stale. Could you use RB (https://reviews.apache.org/r/25238/)?
Thanks Navis. The patch looks great to me. Left a minor question on the review. +1 otherwise.
Overall: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12674889/HIVE-2828.9.patch.txt
ERROR: -1 due to 2 failed/errored test(s), 6559 tests executed
Failed tests:
org.apache.hive.beeline.TestSchemaTool.testSchemaInit org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1280/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1280/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1280/
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed
This message is automatically generated.
ATTACHMENT ID: 12674889
Fails are not related to this. Committed to trunk. Thanks Swarnim for the review!
My bad. Some codes not in the patch is included by mistake, failing hbase_custom_key* tests. I'll merge patch for that with HIVE-8465.
Doc note: This should be documented in the HBase Integration wikidoc, with version information, and a release note would be nice.
navis requested code review of "
HIVE-2828[jira] make timestamp accessible in the hbase KeyValue".Reviewers: JIRA
DPAL-861 make timestamp accessible in the hbase KeyValue
Originated from
HIVE-2781and not accepted, but I think this could be helpful to someone.By using special column notation ':timestamp' in HBASE_COLUMNS_MAPPING, user might access timestamp value in hbase KeyValue.
CREATE TABLE hbase_table (key int, value string, time timestamp)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf:string,:timestamp")
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D1989
AFFECTED FILES
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java
hbase-handler/src/test/queries/hbase_timestamp.q
hbase-handler/src/test/results/hbase_timestamp.q.out
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/4245/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.