[KUDU-1338] Tablet stuck in RaftConfig change currently pending

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.7.0
• Fix Version/s: 0.8.0
• Component/s: consensus
• Target Version/s: 0.8.0
• Code Review: http://gerrit.cloudera.org:8080/#/c/2483/
• Assignee: Todd Lipcon
• Reporter: Jean-Daniel Cryans
• Created: 17/Feb/16 17:50
• Updated: 01/Apr/16 22:02
• Resolved: 01/Apr/16 22:02

Description

We've been adapting the consensus logs for a while and I think we can finally get to the bottom of this issue. I'm attaching the logs from the 3 nodes that participated in the same config for tablet eaa1877a2b3540cf8202aff844c6ca79.

ITBLL is driving the load and eventually fails at 2016-02-15 14:53:12,005 trying to write to node-2 AKA a1081edd2ca24f6b9dcdd7e5000f95ec. The peer that gets stuck is node-5 AKA cdec7fdacbac4ad1b095275b3bdbbe5c, starting from this line:

I0215 14:28:41.585695  2020 raft_consensus_state.cc:459] T eaa1877a2b3540cf8202aff844c6ca79 P cdec7fdacbac4ad1b095275b3bdbbe5c [term 69 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.

The chaos monkey running on this setup is dropping packets one node at time.

I'll attach the logs in a moment.