[CASSANDRA-19018] An SAI-specific mechanism to ensure consistency isn't violated for multi-column (i.e. AND) queries at CL > ONE

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 5.0-rc1, 5.0, 5.1
• Component/s: Consistency/Coordination, Feature/SAI
• Epic Link: Storage Attached Indexes (Phase 2)
• Bug Category: Correctness - Transient Incorrect Response
• Severity: Critical
• Complexity: Challenging
• Discovered By: Adhoc Test
• Platform: All
• Since Version: 5.0-alpha1

Source Control Link

https://github.com/apache/cassandra/commit/f7984627e7dd5794dabc099f49ecf80ca3631803

Test and Documentation Plan

New matrix of almost 200 tests that covers both RFP and the newly discovered consistency problems.

• Assignee: Caleb Rackliffe
• Reporter: Caleb Rackliffe

Authors

Caleb Rackliffe

Reviewers

Alex Petrov, Andres de la Peña

• Created: 09/Nov/23 20:38
• Updated: 07/Mar/24 21:42
• Resolved: 29/Feb/24 21:06

Description

CASSANDRA-19007 is going to be where we add a guardrail around filtering/index queries that use intersection/AND over partially updated non-key columns. (ex. Restricting one clustering column and one normal column does not cause a consistency problem, as primary keys cannot be partially updated.) This issue exists to attempt to fix this specifically for SAI in 5.0.x, as Accord will (last I checked) not be available until the 5.1 release.

The SAI-specific version of the originally reported issue is this:

try (Cluster cluster = init(Cluster.build(2).withConfig(config -> config.with(GOSSIP).with(NETWORK)).start()))
        {
            cluster.schemaChange(withKeyspace("CREATE TABLE %s.t (k int PRIMARY KEY, a int, b int)"));
            cluster.schemaChange(withKeyspace("CREATE INDEX ON %s.t(a) USING 'sai'"));
            cluster.schemaChange(withKeyspace("CREATE INDEX ON %s.t(b) USING 'sai'"));

            // insert a split row
            cluster.get(1).executeInternal(withKeyspace("INSERT INTO %s.t(k, a) VALUES (0, 1)"));
            cluster.get(2).executeInternal(withKeyspace("INSERT INTO %s.t(k, b) VALUES (0, 2)"));

        // Uncomment this line and test succeeds w/ partial writes completed...
        //cluster.get(1).nodetoolResult("repair", KEYSPACE).asserts().success();

            String select = withKeyspace("SELECT * FROM %s.t WHERE a = 1 AND b = 2");
            Object[][] initialRows = cluster.coordinator(1).execute(select, ConsistencyLevel.ALL);
            assertRows(initialRows, row(0, 1, 2)); // not found!!
        }

To make a long story short, the local SAI indexes are hiding local partial matches from the coordinator that would combine there to form full matches. Simple non-index filtering queries also suffer from this problem, but they hide the partial matches in a different way. I'll outline a possible solution for this in the comments that takes advantage of replica filtering protection and the repaired/unrepaired datasets...and attempts to minimize the amount of extra row data sent to the coordinator.

Issue Links

• fixes: CASSANDRA-19459 test_complementary_deletion_with_limit_on_partition_key_column_with_empty_partitions fails with SAI
• is related to: CASSANDRA-8272 2ndary indexes can return stale data
• is related to: CASSANDRA-18994 SAI range query does not play together with "IN"
• relates to: CASSANDRA-19007 Queries with multi-column replica-side filtering can miss rows
• links to: GitHub Pull Request #2935
• links to: GitHub Pull Request #3083