[HBASE-11684] HBase replicationSource should support multithread to ship the log entry

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: Performance, Replication
• Assignee: Unassigned
• Reporter: Tianying Chang
• Due: 15/Aug/14
• Created: 06/Aug/14 04:26
• Updated: 17/Jun/22 06:12
• Resolved: 21/Mar/16 12:38

Description

We found the replication rate cannot keep up with the write rate when the master cluster is write heavy. We got huge log queue build up due to that. But when we do a rolling restart of master cluster, we found that the appliedOpsRate doubled due to the extra thread created to help recover the log of the restarted RS. ReplicateLogEntries is a synchronous blocking call, it becomes the bottleneck when is only runs with one thread. I think we should support multi-thread for the replication source to ship the data. I don't see any consistency problem. Any other concern here?