[HBASE-19064] Synchronous replication for HBase

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1
• Component/s: Replication
• Labels: Replication
• Assignee: Duo Zhang
• Reporter: Duo Zhang
• Created: 22/Oct/17 06:25
• Updated: 29/Apr/19 21:22
• Resolved: 28/Jun/18 10:24

Description

The guys from Alibaba made a presentation on HBaseCon Asia about the synchronous replication for HBase. We(Xiaomi) think this is a very useful feature for HBase so we want to bring it into the community version.

This is a big feature so we plan to do it in a feature branch.

Issue Links

• duplicates: HBASE-9469 Synchronous replication
• is blocked by: HBASE-19397 Design procedures for ReplicationManager to notify peer change event from master
• is related to: HBASE-20422 Synchronous replication for HBase phase II
• links to: Design Doc
• links to: Review Board

Sub-Tasks

• Add a remote peer cluster wal directory config for synchronous replication: Resolved
• Support setting up two clusters with A and S state: Resolved
• Implement a procedure to convert RS from A to DA: Resolved
• Implement a procedure to convert RS from S to DA: Resolved
• Reject read/write from client but accept write from replication in state S: Resolved
• Introduce a new log writer which can write to two HDFSes: Resolved
• Introduce a special WALProvider for synchronous replication: Resolved
• Add a new cluster state flag for synchronous replication: Resolved
• Reject the replication request when peer is DA or A state: Resolved
• Allow temporarily skipping the creation of remote writer for SynchronousReplicationWALProvider: Resolved
• Complete the procedure for adding a sync replication peer: Resolved
• Use protobuf instead of enum.ordinal to store SyncReplicationState: Resolved
• Add UT for sync replication peer in DA state: Resolved
• Only allow table replication for sync replication for now: Resolved
• Introduce a new base class for replication peer procedure: Resolved
• Allow write request from replication but reject write request from user client when S state.: Resolved
• Only allow removing sync replication peer which is in DA state: Resolved
• General framework to transit sync replication state: Resolved
• General framework to transit sync replication state: Resolved
• Implement a procedure to replay sync replication wal for standby cluster: Resolved
• Create remote wal directory when transitting to state S: Resolved
• Remove the SYNC_REPLICATION_ENABLED flag: Resolved
• Forbid major compaction when standby cluster replay the remote wals: Resolved
• Also remove the wal file in remote cluster when we finish replicating a file: Resolved
• Operational documents for synchronous replication.: Resolved
• Do not write the cluster id of the current active cluster when writing remote WAL: Resolved
• Give up replicating anything in S state: Resolved
• Cleanup related resources when remove a sync replication peer: Resolved
• Also remove remote wals when peer is in DA state: Resolved
• Support removing a ReplicationSourceShipper for a special wal group: Resolved
• Support removing a WAL from LogRoller: Resolved
• Concurrency issue for CombinedAsyncWriter: Resolved
• NPE in RecoverStandbyProcedure.execute: Resolved
• Check remote WAL directory when creating peer and transiting peer to A: Resolved
• Allow writing WAL to local and remote cluster concurrently: Resolved
• Polish the WAL switching when transiting from A to S: Resolved
• Reopen regions using ReopenTableRegionsProcedure: Resolved
• Performance comparison of synchronous replication branch and master branch: Resolved
• NPE encountered when rolling update from master with an async peer to branch HBASE-19064: Resolved
• Fix the style issues on branch HBASE-19064 before merging back to master: Resolved
• Fix broken TestReplicationShell: Resolved