[KUDU-1735] CHECK failure when aborting an ignored config change operation

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.1
• Fix Version/s: 1.1.0
• Component/s: consensus
• Target Version/s: 1.1.0
• Code Review: http://gerrit.cloudera.org:8080/4916
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 27/Oct/16 20:04
• Updated: 08/Nov/16 17:36
• Resolved: 08/Nov/16 17:36

Description

The following sequence causes a CHECK failure:

• a tablet server receives a CONFIG_CHANGE operation
• the tablet server commits the operation (writing the new consensus config to disk), but crashes before it can write the associated COMMIT message to the log
• the server is down for long enough that it is removed from the configuration again while it's down
• when it comes back up, it sees the CONFIG_CHANGE again as a pending replicate. When it's added to PendingRounds, it is ignored as we can see that this configuration is already committed.
• the tserver gets the request from the master to DeleteTablet because it's no longer part of the configuration
  • when trying to abort the operation, it fires a CHECK "Aborting CHANGE_CONFIG_OP but there was no pending config set."