[KUDU-139] Timestamps are being ignored on missed deltas and missed delta replay might be broken

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: M3
• Component/s: tablet
• Code Review: http://gerrit.ent.cloudera.com:8080/#/c/1793/
• Assignee: Unassigned
• Reporter: David Alves
• Created: 28/Feb/14 00:55
• Updated: 26/Feb/16 13:10
• Resolved: 16/Jun/14 15:48

Description

This is close to KUDU-138 but even hairier.

We're ignoring the timestamps on the mutations stored as missed deltas (both for r/w). While this is a problem in itself a worse problem is that we have no good way to restore snapshot order on replay even if we did have the timestamps.

In particular missed delta mutations have the timestamp of the original mutation that created them, but because they are stored out-of-order in the log when we replay there is no guarantee that we apply them before any other mutation that mutates the same rows.

To solve this we need to both store the timestamps and keep them on replay and we need to enforce the correct ordering of mutations even if missed deltas come in the log after a transactions that followed them in the tablet.

Illustration

Base->mut1->|flush snap|mut2->|phase 2 snap|mut3

So mutation 1 was merged with the base (or transformed into an UNDO) mutation 2 will be a missed delta and mutation 3 is some other mutation.

assuming each mutation is its own write in the log we might get

W1,W2,MD,W3

on replay we ignore W1 which is in fact flushed, ignore W2 'cause we think it is flushed (even though it isn't) , restore mut2 though MD and then replay mut3 and we get

Base->mut2->mut3 which is correct

However missed delta ordering is not enforced in the log, so instead of W1,W2,MD,W3 we might get:

W1,W2,W3,MD

On replay we ignore W1,W2 apply W3 and then apply the MD which makes us get:

Base->mut3->mut2 which is not correct.

Issue Links

• blocks: KUDU-159 Not checking value of applied_mutation result at end of TabletBootstrap::PlayMissedDeltaUpdates()
• is blocked by: KUDU-154 Flushing even a single delta makes the TS segfault
• is related to: KUDU-144 change tablet-test and mt-tablet-test to run against TabletPeer
• relates to: KUDU-138 Timestamps are being ignored in bootstrap