[HBASE-24666] Offload the replication source/sink job to independent Replication Server

• Type: Umbrella
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Guanghao Zhang
• Created: 01/Jul/20 04:36
• Updated: 30/Mar/21 07:07

Description

The basic idea is add a role "ReplicationServer" to take the replication source/sink job. HMaster is responsible for scheduling the replication job to different ReplicationServer.

link Design doc

Suggestions are welcomed. Thanks.

Issue Links

• is related to: HBASE-25714 Offload the compaction job to independent Compaction Server
• links to: Design doc

Sub-Tasks

• Remove ReplicationSourceInterface#enqueueLog method and get the new WAL to replicate from ReplicationQueueStorage directly: Open
• Add master UI to show ReplicationServer: Open
• Support start/stop replication server by scripts: Open
• Find a way to know whether logQueue goes empty when ReplicationSource is running on ReplicationServer: Open
• Add heartbeat for ReplicationServer: Open
• Dispatch replication sources to ReplicationServer: Open