[KUDU-2048] consensus: Only evict a replica is a majority is up to date

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.6.0
• Component/s: consensus, recovery
• Code Review: http://gerrit.cloudera.org:8080/8245
• Assignee: Todd Lipcon
• Reporter: Mike Percy
• Created: 21/Jun/17 23:08
• Updated: 19/Jan/18 00:46
• Resolved: 19/Jan/18 00:46

Description

In the context of replica eviction and 3-2-3 recovery, we currently have a "hacky" rule that states that evicting down to less than 2 replicas in a config is prohibited. However we don't currently check to see, when evicting, whether that would leave the config with less than a majority of caught-up voters.

That means, for example, that if we have a config of 3 replicas

{ A, B, C }

and B falls behind, so is currently undergoing a tablet copy, and C goes offline then the algorithm will evict C. However, since A is the only up-to-date replica, this leaves the config in a state where nothing can commit until B is done copying. Even worse, if B is killed or has an error, then we are left in a state that requires manual recovery.

Consider adding an additional rule that states that to evict a node, we must have a majority of up-to-date replicas that are recently active. This will help prevent certain problem scenarios like the above from occurring.

Issue Links

• is related to: KUDU-2183 Leaders changing to an unstable/stuck configuration