[ACCUMULO-378] Multi data center replication

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.7.0
• Component/s: replication
• Assignee: Josh Elser
• Reporter: Sapan Shah
• Created: 07/Feb/12 17:16
• Updated: 09/Sep/16 16:09
• Resolved: 14/Jun/14 04:55

Description

The use case here is where people have multiple data centers and need to replicate the data in between them. Accumulo can model this replication after the way that HBase currently handles the replication as detailed here (http://hbase.apache.org/replication.html).

There will be one master Cluster and multiple slave clusters. Accumulo will use the Master-Push model to replicate the statements from the master clusters WAL to the various slaves WALs.

Issue Links

• is related to: ACCUMULO-3737 Create 1.7.0 release notes
• links to: Active Design Doc Review
• links to: Design Document
• links to: Implementation ReviewBoard
• links to: Original design doc link

Sub-Tasks

• Define replication state machines for data: Resolved
• Define ReplicaSystem interface: Resolved
• Define Thrift IDL for data transfer: Resolved
• Define storage data structure for data that needs replication: Resolved
• Update GC to not remove files before replication has happened: Resolved
• Modify "live ingest" code path to create replication entries: Resolved
• Create service in Master to assign replication work: Resolved
• Create insight to replication in the Monitor: Resolved
• Create services that handles replication "receives": Resolved
• Provide authentication criteria between a master and slave: Resolved
• Enable wire encryption for replication: Resolved
• Property additions for configuration of replication: Resolved
• Don't try to assign permissions to !SYSTEM user when creating a table: Resolved
• Create service in Master to manage replication records: Resolved
• Create ReplicationOperations: Resolved
• Provide WorkAssigner which is order-aware: Resolved
• Create and configure a shell formatter for Status messages: Resolved
• Immediately re-queue files which have more data to replicate: Resolved
• Need to re-use DataInputStream for reading files that need replication: Resolved
• Update user documentation: Resolved
• Replication table should be user-controlled or live in accumulo namespace: Resolved
• Clean up early draft versions of thrift replication classes: Resolved