[KUDU-559] Reject/timeout requests when consensus majority is down

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: M5
• Component/s: consensus
• Target Version/s: Public beta
• Assignee: Adar Dembo
• Reporter: Todd Lipcon
• Created: 04/Dec/14 17:12
• Updated: 26/Feb/16 11:48
• Resolved: 13/Aug/15 20:23

Description

Currently, if a node is the leader but the majority of nodes in the quorum have crashed, it still allows writers to submit messages to the consensus queue, and never times them out. Eventually the queue starts filling up and callers will receive "queue full" errors, but the RPCs associated with the messages stuck in the queue never get responded to. Instead, they should be timed out or otherwise responded to more quickly.

We also need to handle the case where an old leader has become partitioned from its quorum. It needs to at some point discover that it hasn't successfully heartbeat in longer than the election timeout, and step down on its own, or else clients may continually access it and not ever detect that they need to go find the new leader.