[KUDU-1354] MVCC Snapshots chosen during flush can contain out-of-order transactions

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.7.0
• Fix Version/s: 0.8.0
• Component/s: tablet
• Target Version/s: 0.8.0
• Code Review: http://gerrit.cloudera.org:8080/#/c/2342/
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 28/Feb/16 01:10
• Updated: 01/Mar/16 07:45
• Resolved: 01/Mar/16 07:45

Description

I spent a while trying to debug a failure of alter_table-randomized-test and found the following interesting logs:

• We have two operations in the WAL which arrived in short succession (about 4ms apart) just before an alter table. I've renumbered the txids for readability here:

  1.13@2	REPLICATE WRITE_OP
  	op 0: MUTATE (int32 key=1643562) SET c6=1107303203
  1.14@4	REPLICATE WRITE_OP
  	op 0: MUTATE (int32 key=1643562) DELETE
  

• and the Flush that was caused by the Altertable has the following snapshots:

  ... Phase 1 snapshot:  MvccSnapshot[committed={T|T < 2 or (T in (4))]
  ...
  ... Phase 2 snapshot: MvccSnapshot[committed={T|T < 2 or (T in (4, 2))]
  

  Note that the first snapshot considers the 'DELETE' committed but not the 'UPDATE'. We then fill in the 'UPDATE' in the second snapshot.The end result here is that we end up flushing REDO deltas as follows:
  REDO file 1 (flushed in phase 1): includes only the DELETE
  REDO file 2 (flushed after ReupdateMissedDeltas); includes only the UPDATE

When we later proceed to compact this rowset, we get "Check failed: !is_deleted Got UPDATE for deleted row."
Scenarios like this seem to reproduce a few tenths of a percent of the time in this stress test.

Issue Links

• relates to: KUDU-1341 Out of order UNDO in delta file, possibly related to REINSERT