Uploaded image for project: 'Accumulo'
  1. Accumulo
  2. ACCUMULO-378

Multi data center replication

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.7.0
    • Component/s: replication
    • Labels:
      None

      Description

      The use case here is where people have multiple data centers and need to replicate the data in between them. Accumulo can model this replication after the way that HBase currently handles the replication as detailed here (http://hbase.apache.org/replication.html).

      There will be one master Cluster and multiple slave clusters. Accumulo will use the Master-Push model to replicate the statements from the master clusters WAL to the various slaves WALs.

        Attachments

          Issue Links

          1.
          Define replication state machines for data Sub-task Resolved Josh Elser  
          2.
          Define ReplicaSystem interface Sub-task Resolved Josh Elser  
          3.
          Define Thrift IDL for data transfer Sub-task Resolved Josh Elser  
          4.
          Define storage data structure for data that needs replication Sub-task Resolved Josh Elser  
          5.
          Update GC to not remove files before replication has happened Sub-task Resolved Josh Elser  
          6.
          Modify "live ingest" code path to create replication entries Sub-task Resolved Josh Elser  
          7.
          Create service in Master to assign replication work Sub-task Resolved Josh Elser  
          8.
          Create insight to replication in the Monitor Sub-task Resolved Josh Elser  
          9.
          Create services that handles replication "receives" Sub-task Resolved Josh Elser  
          10.
          Provide authentication criteria between a master and slave Sub-task Resolved Josh Elser  
          11.
          Enable wire encryption for replication Sub-task Resolved Unassigned  
          12.
          Property additions for configuration of replication Sub-task Resolved Josh Elser  
          13.
          Don't try to assign permissions to !SYSTEM user when creating a table Sub-task Resolved Josh Elser  
          14.
          Create service in Master to manage replication records Sub-task Resolved Josh Elser  
          15.
          Create ReplicationOperations Sub-task Resolved Josh Elser  
          16.
          Provide WorkAssigner which is order-aware Sub-task Resolved Josh Elser  
          17.
          Create and configure a shell formatter for Status messages Sub-task Resolved Josh Elser  
          18.
          Immediately re-queue files which have more data to replicate Sub-task Resolved Josh Elser  
          19.
          Need to re-use DataInputStream for reading files that need replication Sub-task Resolved Josh Elser  
          20.
          Update user documentation Sub-task Resolved Josh Elser  
          21.
          Replication table should be user-controlled or live in accumulo namespace Sub-task Resolved Christopher Tubbs

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          22.
          Clean up early draft versions of thrift replication classes Sub-task Resolved Josh Elser

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 10m

            Activity

              People

              • Assignee:
                elserj Josh Elser
                Reporter:
                sapanbshah42 Sapan Shah
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m