HCatalog
  1. HCatalog
  2. HCATALOG-189

Zookpeeper based revision manager for HBase Storage Driver

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.3
    • Fix Version/s: 0.3
    • Component/s: None

      Description

      A revision manager for HBase would provide revision numbers for each of the record written using HBase output storage driver in place of timestamps. In addition, the revision manager provides capabilities to take snapshots of HBase tables. Snapshots help in achieving repeatable reads which is a requirement for map-reduce paradigm.

      1. revision-mgmt-core-09.patch
        151 kB
        Vandana Ayyalasomayajula

        Issue Links

          Activity

          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2814/
          -----------------------------------------------------------

          (Updated 2011-12-21 00:23:28.109023)

          Review request for hcatalog, Francis Liu and David Capwell.

          Changes
          -------

          Updated the summary and jira number for the review.

          Summary (updated)
          -------

          A revision manager for HBase would provide revision numbers for each of the record written using HBase output storage driver in place of timestamps. In addition, the revision manager provides capabilities to take snapshots of HBase tables. Snapshots help in achieving repeatable reads which is a requirement for map-reduce paradigm. This patch contains all the core classes for the revision management.

          This addresses bug hcatalog-189.
          https://issues.apache.org/jira/browse/hcatalog-189

          Diffs


          /trunk/storage-drivers/hbase/build.xml 1211608
          /trunk/storage-drivers/hbase/if/transaction.thrift PRE-CREATION
          /trunk/storage-drivers/hbase/ivy.xml 1211608
          /trunk/storage-drivers/hbase/ivy/libraries.properties 1211608
          /trunk/storage-drivers/hbase/src/gen-java/org/apache/hcatalog/hbase/snapshot/transaction/thrift/StoreFamilyRevision.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/gen-java/org/apache/hcatalog/hbase/snapshot/transaction/thrift/StoreFamilyRevisionList.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/FamilyRevision.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/IDGenerator.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/PathUtil.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/RevisionManager.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/RevisionManagerFactory.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/TableSnapshot.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/Transaction.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/ZKBasedRevisionManager.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/ZKUtil.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/lock/LockListener.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/lock/ProtocolSupport.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/lock/WriteLock.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/lock/ZNodeName.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/lock/ZooKeeperOperation.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/snapshot/IDGenClient.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/snapshot/TestIDGenerator.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/snapshot/TestRevisionManager.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/snapshot/TestThriftSerialization.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/snapshot/lock/WriteLockTest.java PRE-CREATION
          /trunk/storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/snapshot/lock/ZNodeNameTest.java PRE-CREATION

          Diff: https://reviews.apache.org/r/2814/diff

          Testing (updated)
          -------

          Unit tests added and they pass.

          Thanks,

          Vandana

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2814/ ----------------------------------------------------------- (Updated 2011-12-21 00:23:28.109023) Review request for hcatalog, Francis Liu and David Capwell. Changes ------- Updated the summary and jira number for the review. Summary (updated) ------- A revision manager for HBase would provide revision numbers for each of the record written using HBase output storage driver in place of timestamps. In addition, the revision manager provides capabilities to take snapshots of HBase tables. Snapshots help in achieving repeatable reads which is a requirement for map-reduce paradigm. This patch contains all the core classes for the revision management. This addresses bug hcatalog-189. https://issues.apache.org/jira/browse/hcatalog-189 Diffs /trunk/storage-drivers/hbase/build.xml 1211608 /trunk/storage-drivers/hbase/if/transaction.thrift PRE-CREATION /trunk/storage-drivers/hbase/ivy.xml 1211608 /trunk/storage-drivers/hbase/ivy/libraries.properties 1211608 /trunk/storage-drivers/hbase/src/gen-java/org/apache/hcatalog/hbase/snapshot/transaction/thrift/StoreFamilyRevision.java PRE-CREATION /trunk/storage-drivers/hbase/src/gen-java/org/apache/hcatalog/hbase/snapshot/transaction/thrift/StoreFamilyRevisionList.java PRE-CREATION /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/FamilyRevision.java PRE-CREATION /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/IDGenerator.java PRE-CREATION /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/PathUtil.java PRE-CREATION /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/RevisionManager.java PRE-CREATION /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/RevisionManagerFactory.java PRE-CREATION /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/TableSnapshot.java PRE-CREATION /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/Transaction.java PRE-CREATION /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/ZKBasedRevisionManager.java PRE-CREATION /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/ZKUtil.java PRE-CREATION /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/lock/LockListener.java PRE-CREATION /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/lock/ProtocolSupport.java PRE-CREATION /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/lock/WriteLock.java PRE-CREATION /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/lock/ZNodeName.java PRE-CREATION /trunk/storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/lock/ZooKeeperOperation.java PRE-CREATION /trunk/storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/snapshot/IDGenClient.java PRE-CREATION /trunk/storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/snapshot/TestIDGenerator.java PRE-CREATION /trunk/storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/snapshot/TestRevisionManager.java PRE-CREATION /trunk/storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/snapshot/TestThriftSerialization.java PRE-CREATION /trunk/storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/snapshot/lock/WriteLockTest.java PRE-CREATION /trunk/storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/snapshot/lock/ZNodeNameTest.java PRE-CREATION Diff: https://reviews.apache.org/r/2814/diff Testing (updated) ------- Unit tests added and they pass. Thanks, Vandana
          Hide
          Francis Liu added a comment -

          made a small change in build.xml for -D testcase.

          Committed in trunk, thanks Vandana.

          Show
          Francis Liu added a comment - made a small change in build.xml for -D testcase. Committed in trunk, thanks Vandana.
          Hide
          Alan Gates added a comment -

          Issue closed with 0.4 release.

          Show
          Alan Gates added a comment - Issue closed with 0.4 release.

            People

            • Assignee:
              Vandana Ayyalasomayajula
              Reporter:
              Vandana Ayyalasomayajula
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development