[CASSANDRA-15071] Materialized View Inconsistent With Base Table Update After Migrating To New DC

• Type: Bug
• Status: Open
• Priority: High
• Resolution: Unresolved
• Component/s: Consistency/Bootstrap and Decommission, Feature/Materialized Views
• Labels: cassandra materializedviews rebuilding
• Bug Category: Correctness
• Severity: Normal
• Complexity: Challenging
• Discovered By: User Report
• Assignee: Unassigned
• Reporter: Avraham Kalvo
• Created: 01/Apr/19 11:21
• Updated: 11/Nov/19 13:33

Description

We've recently completed a successful migration between two data centers in our Cassandra cluster.

After adding the new DC nodes onto the existing cluster, and setting the keyspaces to replicate to both DCs and rebuilding the new DC nodes from the old one, we've cut-over the applications using those keyspaces o start using the new DC exclusively by connecting to its end-points and performing `LOCAL_` consistency level requests there (DCAwareRoundRobinPolicy on LOCAL DC).

We noticed that once the apps started to read data from the materialized views in the new DC, that an inconsistency emerged, which wasn't there in the original DC from which we've migrated.
I.e - source/old DC had the materialized view reflecting the column update on the base table, while target/new DC didn't (the column value in the MV remained the same as it was in the base table, prior to the update).

We only found out about it being logged with a support ticket, and for now, mitigated it by simply recreating the materialized view.

Looking for a root cause for such behavior, is this expected, is this somewhat of a requirement which wasn't fulfilled yet for the MV mechanism, such as the ones mentioned in CASSANDRA-13826?

Thanks,
Avi K