[KUDU-152] Log GC race between Append() and Apply()

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: M3
• Component/s: log
• Code Review: http://gerrit.ent.cloudera.com:8080/#/c/2072/
• Assignee: Mike Percy
• Reporter: Mike Percy
• Created: 06/Mar/14 04:31
• Updated: 26/Feb/16 13:25
• Resolved: 14/May/14 15:07

Description

Currently in the OpId anchoring Log GC code, there is a race such that through a series of particular events we may delete WAL files before they are persisted into the in-memory data structures, where they will normally be anchored until flush to disk.

This is the series of events that may occur:

1. Start with a quiesced tablet (no outstanding unflushed data, no log anchors)
2. Log OpId 1 (add OpId 1 to Prepare queue)
3. Log OpId 2, which causes a roll of OpId 1's log segment due to full WAL file
4. Log::GC() thread runs, which sees no anchor on the rolled segment. Delete OpId 1 log segment
5. Prepare for OpId 1 finally completes
6. Apply for OpId 1 finally completes. At this time OpId 1 is anchored in the mem store, however the data is gone.

Due to this race, we need a way to anchor on the OpId as soon as the OpId is assigned inside of Append(), and that anchor must be maintained until Apply() completes.

There are other ideas involving the use of MVCC snapshots to detect such problems, however that only reduces the race window, because in-flight timestamps are not tracked until the Prepare() phase is complete, which is too late in the above scenario. This may further be improved by making mem stores inherit the anchor from their predecessor until some criteria is met (either time-based or some other event), however at this time we don't have consensus on a criteria that will guarantee 100% safety from data loss.

Issue Links

• is blocked by: KUDU-151 Track in-flight Transactions