[KUDU-2763] Confusing "log matching property violated" message on new tablets

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.10.0
• Component/s: consensus, supportability
• Labels: newbie
• Assignee: Mitch Barnett
• Reporter: Todd Lipcon
• Created: 04/Apr/19 15:31
• Updated: 15/May/19 10:12
• Resolved: 15/May/19 10:12

Description

I've seen several operators get confused by the following log message:

I0404 04:31:48.351042 27049 raft_consensus.cc:1053] T ec01a78d890847fa9a7e2c684caf89e3 P 50d64a444f1b409fa69c9566f3913a9c [term 1 FOLLOWER]: Refusing update from remote peer 48ca5ed87b034141a600a21b845a8ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)

This happens on every new tablet, since for whatever reason, the first leader sends its first message with "preceding opid" set to 1.1 instead of 0.0. We could special case this situation and not log it in the replica, but we could also try to get the initial leader to send with preceding_opid=0.0 and avoid an extra consensus round before a new tablet becomes writable. Likely just silencing the log is less risky, since the cost of the extra round is negligible.