navis requested code review of "HIVE-2828 [jira] make timestamp accessible in the hbase KeyValue".
Reviewers: JIRA

DPAL-861 make timestamp accessible in the hbase KeyValue

Originated from HIVE-2781 and not accepted, but I think this could be helpful to someone.

By using special column notation ':timestamp' in HBASE_COLUMNS_MAPPING, user might access timestamp value in hbase KeyValue.

CREATE TABLE hbase_table (key int, value string, time timestamp)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf:string,:timestamp")

TEST PLAN
EMPTY

REVISION DETAIL
https://reviews.facebook.net/D1989

AFFECTED FILES
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java
hbase-handler/src/test/queries/hbase_timestamp.q
hbase-handler/src/test/results/hbase_timestamp.q.out

MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/4245/

Tip: use the X-Herald-Rules header to filter Herald messages in your client.

Does HBase integration in Hive allows retrieval of multiple versions ( diff time stamp ) of the same cell?
Also whether allows specifying the time range while select such that the cell value versions within this time range will be retrieved?

No and no as far as I know, but might be implemented by modifying HiveHBaseTableInputFormat and HBaseStorageHandler.

navis updated the revision "HIVE-2828 [jira] make timestamp accessible in the hbase KeyValue".
Reviewers: JIRA

Applied HIVE-2771

REVISION DETAIL
https://reviews.facebook.net/D1989

AFFECTED FILES
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java.rej
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java.rej
hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java
hbase-handler/src/test/queries/external_table_ppd.q
hbase-handler/src/test/queries/hbase_timestamp.q
hbase-handler/src/test/results/external_table_ppd.q.out
hbase-handler/src/test/results/hbase_timestamp.q.out

navis updated the revision "HIVE-2828 [jira] make timestamp accessible in the hbase KeyValue".
Reviewers: JIRA

1. Rebased on trunk
2. Support string type for timestamp column

REVISION DETAIL
https://reviews.facebook.net/D1989

AFFECTED FILES
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java
hbase-handler/src/test/queries/hbase_timestamp.q
hbase-handler/src/test/results/hbase_timestamp.q.out

navis updated the revision "HIVE-2828 [jira] make timestamp accessible in the hbase KeyValue".
Reviewers: JIRA

Rebased on trunk

REVISION DETAIL
https://reviews.facebook.net/D1989

AFFECTED FILES
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java
hbase-handler/src/test/queries/hbase_timestamp.q
hbase-handler/src/test/results/hbase_timestamp.q.out

navis updated the revision "HIVE-2828 [jira] make timestamp accessible in the hbase KeyValue".
Reviewers: JIRA

Rebased on trunk

REVISION DETAIL
https://reviews.facebook.net/D1989

AFFECTED FILES
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java
hbase-handler/src/test/queries/hbase_timestamp.q
hbase-handler/src/test/results/hbase_timestamp.q.out

Would this allow aggregation across timestamps as well?

If what you meant was is it possible to 'group by timestamp', it's possible.

What is the right process for applying these patches? Also, on which SVN paths would it be applicable? Obviously the code has changed since these were created so I'm trying to figure out how to apply them. We currently have a version of 0.8.0 that would be great to apply these patches against. I tried against branches/0.8.0 to no avail.

Thanks!

John

What is the status of this ticket?
Can someone tell me if this is going to be pushed? or any reason why its not pushed?

This is a good feature and we have come to a couple of situations where this support is good to have.

The current patch might not be applicable to the current version Hive 0.13, but I can work on it (in-fact I have a patch available with the current trunk but slightly different implementation).

2.5 year rebase.

Overall: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12665836/HIVE-2828.6.patch.txt

ERROR: -1 due to 4 failed/errored test(s), 6134 tests executed
Failed tests:

org.apache.hadoop.hive.hbase.TestLazyHBaseObject.testLazyHBaseRow1
org.apache.hadoop.hive.hbase.TestLazyHBaseObject.testLazyHBaseRow2
org.apache.hadoop.hive.hbase.TestLazyHBaseObject.testLazyHBaseRow3
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/595/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/595/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-595/

Messages:

Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed

This message is automatically generated.

ATTACHMENT ID: 12665836

Fixed test fails

Overall: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12665849/HIVE-2828.7.patch.txt

ERROR: -1 due to 1 failed/errored test(s), 6135 tests executed
Failed tests:

org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/598/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/598/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-598/

Messages:

Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed

This message is automatically generated.

ATTACHMENT ID: 12665849

Rebased to trunk.

Overall: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12672261/HIVE-2828.8.patch.txt

SUCCESS: +1 6488 tests passed

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1072/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1072/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1072/

Messages:

Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase

This message is automatically generated.

ATTACHMENT ID: 12672261

Sushanth Sowmyan I think there are some requests on this. Could you review this if time allows?

Sure, I'll try to look into this tonight.

Sorry, I've been swamped the last couple of days, and expect to be so for the next couple of days - this weekend is going to be the earliest I can look at it. I think Brock Noland and Nick Dimiduk also potentially have some context, so tagging them if they want to look at this before I do so.

Can you give a github link where to look at?

Swarnim Kulkarni has worked on the hbase serde lately.

Ok,thanks!

Sushanth Sowmyan, Brock Noland, Swarnim Kulkarni, Sergey
Also, HIVE-8267 is the sequel feature request, with the ability to make timestamp of any cell available. So could you guys also look into that?

Sure. I will take a look at it today.

Left comments on the phabricator review.

Left a comment on phabricator. +1

ndimiduk added a subscriber: ndimiduk.
ndimiduk accepted this revision.
ndimiduk added a reviewer: ndimiduk.
ndimiduk added a comment.
This revision is now accepted and ready to land.

+1

BRANCH
DPAL-861

INLINE COMMENTS
hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java:154 This is as good a decision as any. Maybe down the road someone will want to specify which cell's timestamp maps to which column?

REVISION DETAIL
https://reviews.facebook.net/D1989

To: navis, JIRA, ndimiduk
Cc: ndimiduk

Nick Dimiduk It would be handled in HIVE-8267. (see https://issues.apache.org/jira/browse/HIVE-8267?focusedCommentId=14154347&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14154347)
Swarnim Kulkarni Cannot find your comments in phabricator and the patch in it would have gone stale. Could you use RB (https://reviews.apache.org/r/25238/)?

Thanks Navis. The patch looks great to me. Left a minor question on the review. +1 otherwise.

Addressed comment. Rerunning test.

Overall: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12674889/HIVE-2828.9.patch.txt

ERROR: -1 due to 2 failed/errored test(s), 6559 tests executed
Failed tests:

org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1280/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1280/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1280/

Messages:

Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed

This message is automatically generated.

ATTACHMENT ID: 12674889

Fails are not related to this. Committed to trunk. Thanks Swarnim for the review!

My bad. Some codes not in the patch is included by mistake, failing hbase_custom_key* tests. I'll merge patch for that with HIVE-8465.

Doc note: This should be documented in the HBase Integration wikidoc, with version information, and a release note would be nice.

• HBase Integration