Hive
  1. Hive
  2. HIVE-2781

HBaseSerDe should allow users to specify the timestamp passed to Puts

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.9.0
    • Component/s: None
    • Labels:
      None
    • Tags:
      hbase

      Description

      Users may want to specify the timestamp used for Put requests to hbase. Thus enabling users to have the same timestamp for a single batch of writes. Which would be useful for a number of things. HCatalog's HBase storageHandler implementation makes use of this feature to provide users with snapshot isolation and write transactions. My proposal is to add the timestamp option as a final static member:

      public static final long HBASE_PUT_TIMESTAMP = "hbase.put_timestamp"

      And passing this value to all the Puts created by serialize()

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-2781: HBaseSerDe should allow users to specify the timestamp passed to Puts (toffer via hashutosh) (Revision 1293616)

        Result = ABORTED
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293616
        Files :

        • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
        • /hive/trunk/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2781 : HBaseSerDe should allow users to specify the timestamp passed to Puts (toffer via hashutosh) (Revision 1293616) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293616 Files : /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java /hive/trunk/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java
        Hide
        Ashutosh Chauhan added a comment -

        This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

        Show
        Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
        Hide
        Phabricator added a comment -

        navis has abandoned the revision "HIVE-2781 [jira] HBaseSerDe should allow users to specify the timestamp passed to Puts".

        Not accepted

        REVISION DETAIL
        https://reviews.facebook.net/D1863

        Show
        Phabricator added a comment - navis has abandoned the revision " HIVE-2781 [jira] HBaseSerDe should allow users to specify the timestamp passed to Puts". Not accepted REVISION DETAIL https://reviews.facebook.net/D1863
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1276 (See https://builds.apache.org/job/Hive-trunk-h0.21/1276/)
        HIVE-2781: HBaseSerDe should allow users to specify the timestamp passed to Puts (toffer via hashutosh) (Revision 1293616)

        Result = FAILURE
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293616
        Files :

        • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
        • /hive/trunk/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1276 (See https://builds.apache.org/job/Hive-trunk-h0.21/1276/ ) HIVE-2781 : HBaseSerDe should allow users to specify the timestamp passed to Puts (toffer via hashutosh) (Revision 1293616) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293616 Files : /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java /hive/trunk/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java
        Hide
        Ashutosh Chauhan added a comment -

        Committed to trunk. Thanks, Francis!

        Show
        Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Francis!
        Hide
        Francis Liu added a comment -

        license granted

        Show
        Francis Liu added a comment - license granted
        Hide
        Phabricator added a comment -

        ashutoshc has accepted the revision "HIVE-2781 [jira] HBaseSerDe should allow users to specify the timestamp passed to Puts".

        +1 Running tests.

        REVISION DETAIL
        https://reviews.facebook.net/D1881

        BRANCH
        HBaseSerDe

        Show
        Phabricator added a comment - ashutoshc has accepted the revision " HIVE-2781 [jira] HBaseSerDe should allow users to specify the timestamp passed to Puts". +1 Running tests. REVISION DETAIL https://reviews.facebook.net/D1881 BRANCH HBaseSerDe
        Hide
        Ashutosh Chauhan added a comment -

        @Francis,
        Can you upload the patch on jira granting licensing rights?

        Show
        Ashutosh Chauhan added a comment - @Francis, Can you upload the patch on jira granting licensing rights?
        Hide
        Francis Liu added a comment -

        You submitted a patch first. I just wanted to point out that your implementation was not what I had in mind when I filed this Jira.

        Show
        Francis Liu added a comment - You submitted a patch first. I just wanted to point out that your implementation was not what I had in mind when I filed this Jira.
        Hide
        Navis added a comment -

        I didn't know that there was already a patch, sorry.

        Show
        Navis added a comment - I didn't know that there was already a patch, sorry.
        Hide
        Francis Liu added a comment -

        Navis, my intention was a bit different when I filed this jira. I wouldn't want to expose the timestamp field to users as we are using it internally for dealing with concurrent writes and reads on a table. See attached patch. Though I can a use case for your patch as well, not just what I had in mind for this Jira .

        Show
        Francis Liu added a comment - Navis, my intention was a bit different when I filed this jira. I wouldn't want to expose the timestamp field to users as we are using it internally for dealing with concurrent writes and reads on a table. See attached patch. Though I can a use case for your patch as well, not just what I had in mind for this Jira .
        Hide
        Phabricator added a comment -

        toffer requested code review of "HIVE-2781 [jira] HBaseSerDe should allow users to specify the timestamp passed to Puts".
        Reviewers: JIRA

        Users may want to specify the timestamp used for Put requests to hbase. Thus enabling users to have the same timestamp for a single batch of writes. Which would be useful for a number of things. HCatalog's HBase storageHandler implementation makes use of this feature to provide users with snapshot isolation and write transactions. My proposal is to add the timestamp option as a final static member: public static final long HBASE_PUT_TIMESTAMP = "hbase.put_timestamp". And passing this value to all the Puts created by serialize()

        TEST PLAN
        Added an unit test.

        REVISION DETAIL
        https://reviews.facebook.net/D1881

        AFFECTED FILES
        hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
        hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/4005/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - toffer requested code review of " HIVE-2781 [jira] HBaseSerDe should allow users to specify the timestamp passed to Puts". Reviewers: JIRA Users may want to specify the timestamp used for Put requests to hbase. Thus enabling users to have the same timestamp for a single batch of writes. Which would be useful for a number of things. HCatalog's HBase storageHandler implementation makes use of this feature to provide users with snapshot isolation and write transactions. My proposal is to add the timestamp option as a final static member: public static final long HBASE_PUT_TIMESTAMP = "hbase.put_timestamp". And passing this value to all the Puts created by serialize() TEST PLAN Added an unit test. REVISION DETAIL https://reviews.facebook.net/D1881 AFFECTED FILES hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/4005/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Phabricator added a comment -

        navis updated the revision "HIVE-2781 [jira] HBaseSerDe should allow users to specify the timestamp passed to Puts".
        Reviewers: JIRA

        Bug fix for timestamp=-1

        REVISION DETAIL
        https://reviews.facebook.net/D1863

        AFFECTED FILES
        hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
        hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
        hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
        hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java
        hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java
        hbase-handler/src/test/queries/hbase_timestamp.q
        hbase-handler/src/test/results/hbase_timestamp.q.out

        Show
        Phabricator added a comment - navis updated the revision " HIVE-2781 [jira] HBaseSerDe should allow users to specify the timestamp passed to Puts". Reviewers: JIRA Bug fix for timestamp=-1 REVISION DETAIL https://reviews.facebook.net/D1863 AFFECTED FILES hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java hbase-handler/src/test/queries/hbase_timestamp.q hbase-handler/src/test/results/hbase_timestamp.q.out
        Hide
        Phabricator added a comment -

        navis has commented on the revision "HIVE-2781 [jira] HBaseSerDe should allow users to specify the timestamp passed to Puts".

        Added special column :timestamp to interchange timestamp with hbase value. This is only a suggestion and should do more works including test and javadocs.

        REVISION DETAIL
        https://reviews.facebook.net/D1863

        Show
        Phabricator added a comment - navis has commented on the revision " HIVE-2781 [jira] HBaseSerDe should allow users to specify the timestamp passed to Puts". Added special column :timestamp to interchange timestamp with hbase value. This is only a suggestion and should do more works including test and javadocs. REVISION DETAIL https://reviews.facebook.net/D1863
        Hide
        Phabricator added a comment -

        navis requested code review of "HIVE-2781 [jira] HBaseSerDe should allow users to specify the timestamp passed to Puts".
        Reviewers: JIRA

        DPAL-861 HBaseSerDe should allow users to specify the timestamp passed to Puts

        Users may want to specify the timestamp used for Put requests to hbase. Thus enabling users to have the same timestamp for a single batch of writes. Which would be useful for a number of things. HCatalog's HBase storageHandler implementation makes use of this feature to provide users with snapshot isolation and write transactions. My proposal is to add the timestamp option as a final static member:

        public static final long HBASE_PUT_TIMESTAMP = "hbase.put_timestamp"

        And passing this value to all the Puts created by serialize()

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D1863

        AFFECTED FILES
        hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
        hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
        hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
        hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java
        hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java
        hbase-handler/src/test/queries/hbase_timestamp.q
        hbase-handler/src/test/results/hbase_timestamp.q.out

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/3957/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - navis requested code review of " HIVE-2781 [jira] HBaseSerDe should allow users to specify the timestamp passed to Puts". Reviewers: JIRA DPAL-861 HBaseSerDe should allow users to specify the timestamp passed to Puts Users may want to specify the timestamp used for Put requests to hbase. Thus enabling users to have the same timestamp for a single batch of writes. Which would be useful for a number of things. HCatalog's HBase storageHandler implementation makes use of this feature to provide users with snapshot isolation and write transactions. My proposal is to add the timestamp option as a final static member: public static final long HBASE_PUT_TIMESTAMP = "hbase.put_timestamp" And passing this value to all the Puts created by serialize() TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1863 AFFECTED FILES hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java hbase-handler/src/test/queries/hbase_timestamp.q hbase-handler/src/test/results/hbase_timestamp.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/3957/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          People

          • Assignee:
            Francis Liu
            Reporter:
            Francis Liu
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development