[KUDU-821] Tight loop trying to send requests to dead server after logs GCed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Private Beta
• Fix Version/s: Public beta
• Component/s: consensus
• Target Version/s: Public beta
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 11/Jun/15 13:56
• Updated: 29/Feb/16 09:23
• Resolved: 08/Oct/15 02:10

Description

We saw this issue take down a server on bolt after one of its followers died. Eventually, we GCed that follower's logs, and then we started logging about 25 times a second that we couldn't GC its logs.

This also caused a lot of lock contention trying to write to other peers, etc, and make consensus more or less grind to halt.

Issue Links

• relates to: KUDU-820 Add metrics for diagnosing recent cluster issues