[KUDU-1233] Missing ChangeConfigRequest COMMIT messages in WAL

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.6.0
• Component/s: consensus
• Target Version/s: 0.6.0
• Code Review: http://gerrit.cloudera.org:8080/1366
• Assignee: Mike Percy
• Reporter: Mike Percy
• Created: 26/Oct/15 20:54
• Updated: 17/Nov/15 00:24
• Resolved: 17/Nov/15 00:24

Description

frew hit a case where there is a committed change config request in the consensus metadata but the COMMIT entry in the WAL is missing. Actually, he noticed it because there were two consecutive COMMIT entries missing (both for change config requests) and tablet startup failed because that is an illegal state to be in (one change config request must commit before the next one starts, per the Raft specification).

I am working on reproducing this, as well as providing a tool to allow for insertion of arbitrary data into a WAL in order to recover from such bugs by generating the COMMIT messages needed to complete tablet bootstrap.