[KUDU-1678] Race during abort of pending operations during raft shutdown

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0, 1.6.0, 1.8.0
• Fix Version/s: 1.9.0
• Component/s: consensus
• Assignee: William Berkeley
• Reporter: Todd Lipcon
• Created: 04/Oct/16 20:46
• Updated: 26/Oct/18 16:31
• Resolved: 26/Oct/18 16:31

Description

I'm seeing the following race occasionally in alter_table-randomized-test:

• a follower tablet is shutting down while some operations are pending. The first operation is an ALTER_TABLE, and the second is a WRITE which depends on the ALTER (i.e includes the new column)
• we cancel the ALTER successfully, and then the thread gets de-scheduled
• the PrepareTask for the WRITE runs before we're able to cancel it. It then fails to prepare because the alter it depends on has not completed

It seems like we should probably cancel the pending operations in reverse order.