[CASSANDRA-19018] An SAI-specific mechanism to ensure consistency isn't violated for multi-column (i.e. AND) queries at CL > ONE - ASF JIRA

Agile Board

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Normal
Resolution: Fixed
Fix Version/s: 5.0-rc1, 5.0, 5.1
Component/s: Consistency/Coordination, Feature/SAI
Labels:
None

Epic Link:
Storage Attached Indexes (Phase 2)
Bug Category:
Correctness - Transient Incorrect Response
Severity:
Critical
Complexity:
Challenging
Discovered By:
Adhoc Test
Platform:

All
Impacts:

None
Since Version:

5.0-alpha1
Source Control Link:

https://github.com/apache/cassandra/commit/f7984627e7dd5794dabc099f49ecf80ca3631803
Test and Documentation Plan:

Hide

New matrix of almost 200 tests that covers both RFP and the newly discovered consistency problems.

Show
New matrix of almost 200 tests that covers both RFP and the newly discovered consistency problems.

Description

CASSANDRA-19007 is going to be where we add a guardrail around filtering/index queries that use intersection/AND over partially updated non-key columns. (ex. Restricting one clustering column and one normal column does not cause a consistency problem, as primary keys cannot be partially updated.) This issue exists to attempt to fix this specifically for SAI in 5.0.x, as Accord will (last I checked) not be available until the 5.1 release.

The SAI-specific version of the originally reported issue is this:

try (Cluster cluster = init(Cluster.build(2).withConfig(config -> config.with(GOSSIP).with(NETWORK)).start()))
        {
            cluster.schemaChange(withKeyspace("CREATE TABLE %s.t (k int PRIMARY KEY, a int, b int)"));
            cluster.schemaChange(withKeyspace("CREATE INDEX ON %s.t(a) USING 'sai'"));
            cluster.schemaChange(withKeyspace("CREATE INDEX ON %s.t(b) USING 'sai'"));

            // insert a split row
            cluster.get(1).executeInternal(withKeyspace("INSERT INTO %s.t(k, a) VALUES (0, 1)"));
            cluster.get(2).executeInternal(withKeyspace("INSERT INTO %s.t(k, b) VALUES (0, 2)"));

        // Uncomment this line and test succeeds w/ partial writes completed...
        //cluster.get(1).nodetoolResult("repair", KEYSPACE).asserts().success();

            String select = withKeyspace("SELECT * FROM %s.t WHERE a = 1 AND b = 2");
            Object[][] initialRows = cluster.coordinator(1).execute(select, ConsistencyLevel.ALL);
            assertRows(initialRows, row(0, 1, 2)); // not found!!
        }

To make a long story short, the local SAI indexes are hiding local partial matches from the coordinator that would combine there to form full matches. Simple non-index filtering queries also suffer from this problem, but they hide the partial matches in a different way. I'll outline a possible solution for this in the comments that takes advantage of replica filtering protection and the repaired/unrepaired datasets...and attempts to minimize the amount of extra row data sent to the coordinator.