[CASSANDRA-9995] Add background consistency mode for MV

• Type: New Feature
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Component/s: Feature/Materialized Views
• Assignee: Unassigned
• Reporter: Carl Yeksigian
• Created: 05/Aug/15 17:08
• Updated: 16/Apr/19 09:31

Description

Currently, we only support a fast refresh mode which slows down writes, but brings the base and view to consistency quickly. It would be possible to keep reads and writes close to the same performance they have now by sacrificing the time to consistency.

The way this mode would work is:

• When data is flushed, the sstable is marked as inconsistent for MV
• Compaction can only run on either the set of sstables which are consistent, or the set of sstables which are inconsistent, but cannot mix the two
• A background job would take the sstables which are inconsistent and compare them to the current set which are consistent and generate the appropriate updates for the index to bring it up to date
• Any newly streamed sstables would be marked as inconsistent and would be included the next time the job ran

The background consistency job could be configured to run whenever a new sstable is flushed, or at certain time intervals.

By switching to a job which only looked at the flushed sstables, we wouldn't have to worry about the memtable updates which generate updates to the view but aren't recorded anywhere. We also wouldn't have to do any coordination at write time, use the batchlog for these writes, or issue any new updates when applying the MV update.