[KUDU-915] Bootstrap can fail shortly after an alter-table

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Private Beta
• Fix Version/s: 0.5.0
• Component/s: tablet
• Target Version/s: 1.2.0
• Assignee: Unassigned
• Reporter: Todd Lipcon
• Created: 31/Jul/15 11:42
• Updated: 07/Apr/16 20:50
• Resolved: 07/Apr/16 20:50

Description

I saw a test failure which seems to be due to the following sequence:

1) Log: REPLICATE 1.8 ALTER_SCHEMA
2) Log: REPLICATE 1.9 WRITE
3) Log: COMMIT 1.9 WRITE
4) TabletMetadata::Flush()
5) crash (before COMMIT 1.8 ALTER_SCHEMA)

During bootstrap, we then have an issue that, because we haven't seen a commit message for 1.8, we consider operation 1.9 to be still pending. We are relying on the tablet peer's FlushInFlightsToLogCallback to ensure that we don't flush metadata until the COMMIT message in the log, but that isn't strong enough – we need to actually wait until COMMIT messages are in the log for all prior operations, not just all prior writes. The implementation currently uses MvccManager::WaitForAllInFlightToCommit, but since AlterSchema doesn't use MvccManager, we aren't waiting for it.

Issue Links

• relates to: KUDU-304 Support ALTER_TABLE transactions in consensus replicas