[CASSANDRA-19007] Queries with multi-column replica-side filtering can miss rows

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 5.x
• Component/s: Consistency/Coordination
• Bug Category: Correctness - Consistency
• Severity: Normal
• Complexity: Challenging
• Discovered By: Code Inspection
• Platform: All
• Since Version: 0.3
• Assignee: Caleb Rackliffe
• Reporter: Andres de la Peña

Authors

Caleb Rackliffe

• Created: 07/Nov/23 17:28
• Updated: 01/Oct/24 22:14

Description

SELECT queries with multi-column replica-side filtering can miss rows if the filtered columns are spread across out-of-sync replicas. This dtest reproduces the issue:

@Test
public void testMultiColumnReplicaSideFiltering() throws IOException
{
    try (Cluster cluster = init(Cluster.build().withNodes(2).start()))
    {
        cluster.schemaChange(withKeyspace("CREATE TABLE %s.t (k int PRIMARY KEY, a int, b int)"));

        // insert a split row
        cluster.get(1).executeInternal(withKeyspace("INSERT INTO %s.t(k, a) VALUES (0, 1)"));
        cluster.get(2).executeInternal(withKeyspace("INSERT INTO %s.t(k, b) VALUES (0, 2)"));

        String select = withKeyspace("SELECT * FROM %s.t WHERE a = 1 AND b = 2 ALLOW FILTERING");
        Object[][] initialRows = cluster.coordinator(1).execute(select, ALL);
        assertRows(initialRows, row(0, 1, 2)); // not found!!
    }
}

This edge case affects queries using ALLOW FILTERING or any index implementation.

It affects all branches since multi-column replica-side filtering queries were introduced, long before 3.0.

The protection mechanism added by CASSANDRA-8272/8273 won't deal with this case, since it only solves single-column conflicts where stale rows could resurrect. This bug however doesn't resurrect data, it can only miss rows while the replicas are out-of-sync.

Issue Links

• is related to: CASSANDRA-8272 2ndary indexes can return stale data
• is related to: CASSANDRA-8273 Allow filtering queries can return stale data
• is related to: CASSANDRA-19017 Ensure that empty SAI column indexes do not fail on validation after full-SSTable streaming
• is related to: CASSANDRA-19018 An SAI-specific mechanism to ensure consistency isn't violated for multi-column (i.e. AND) queries at CL > ONE
• is related to: CASSANDRA-7168 Add repair aware consistency levels
• is required by: CASSANDRA-18473 Storage Attached Indexes (Phase 2)
• links to: GitHub Pull Request #3155