[HBASE-2223] Handle 10min+ network partitions between clusters

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.90.0
• Component/s: Replication
• Labels: replication
• Hadoop Flags: Reviewed
• Assignee: Jean-Daniel Cryans
• Reporter: Jean-Daniel Cryans
• Created: 12/Feb/10 19:56
• Updated: 06/May/19 21:03
• Resolved: 01/Jul/10 00:28

Description

We need a nice way of handling long network partitions without impacting a master cluster (which pushes the data). Currently it will just retry over and over again.

I think we could:

• Stop replication to a slave cluster if it didn't respond for more than 10 minutes
• Keep track of the duration of the partition
• When the slave cluster comes back, initiate a MR job like HBASE-2221

Maybe we want less than 10 minutes, maybe we want this to be all automatic or just the first 2 parts. Discuss.

Issue Links

• is blocked by: HBASE-2707 Can't recover from a dead ROOT server if any exceptions happens during log splitting
• is blocked by: HBASE-2539 Cannot start ZK before the rest in tests anymore
• is blocked by: HBASE-2735 Make HBASE-2694 replication-friendly
• is depended upon by: HBASE-2611 Handle RS that fails while processing the failure of another one
• is related to: HBASE-2791 Stop dumping exceptions coming from ZK and do nothing about them
• is related to: HBASE-2809 Accounting of ReplicationSource's memory usage
• is related to: HBASE-2808 Document the implementation of replication
• is related to: HBASE-2810 Profiling of ReplicationSource to determine if it's better to reuse lists or not
• requires: HBASE-2529 Make OldLogsCleaner easier to extend
• requires: HBASE-2527 Add the ability to easily extend some HLog actions
• requires: HBASE-2534 Recursive deletes and misc improvements to ZKW