[CASSANDRA-19011] Primary key -> row ID lookups are broken for skipping and intersections during SAI queries

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 5.0-beta1, 5.0, 5.1
• Component/s: Feature/SAI
• Epic Link: Storage Attached Indexes (Phase 2)
• Bug Category: Availability
• Severity: Critical
• Complexity: Normal
• Discovered By: Fuzz Test
• Platform: All
• Since Version: 5.0-alpha1

Source Control Link

https://github.com/apache/cassandra/commit/6a7bef12ecdf59e3a67c81b89c13e3c2bf7e19d8

Test and Documentation Plan

CI results are in the comments

• Assignee: Mike Adamson
• Reporter: Alex Petrov

Authors

Mike Adamson

Reviewers

Caleb Rackliffe, Michael Semb Wever

• Created: 08/Nov/23 19:57
• Updated: 01/Dec/23 12:56
• Resolved: 01/Dec/23 12:56

Description

Schema:

CREATE TABLE IF NOT EXISTS distributed_test_keyspace.tbl1 (pk1 bigint,ck1 bigint,v1 ascii,v2 bigint, PRIMARY KEY (pk1, ck1)) WITH  CLUSTERING ORDER BY (ck1 ASC);
CREATE CUSTOM INDEX v1_sai_idx ON distributed_test_keyspace.tbl1 (v1) USING 'StorageAttachedIndex' WITH OPTIONS = {'case_sensitive': 'false', 'normalize': 'true', 'ascii': 'true'}; ;
CREATE CUSTOM INDEX v2_sai_idx ON distributed_test_keyspace.tbl1 (v2) USING 'StorageAttachedIndex';
 

java.lang.AssertionError: skipped to an item smaller than the target; iterator: org.apache.cassandra.index.sai.disk.IndexSearchResultIterator@f399f79, target key: PrimaryKey: { token: 8384965201802291970, partition: DecoratedKey(8384965201802291970, c4bc1c50f9e76a50), clustering: CLUSTERING:8b4b4c5991a4ea10 } , returned key: PrimaryKey: { token: 8384965201802291970, partition: DecoratedKey(8384965201802291970, c4bc1c50f9e76a50), clustering: CLUSTERING:89f1cf92658cb668 } 
	at org.apache.cassandra.index.sai.iterators.KeyRangeIntersectionIterator.computeNext(KeyRangeIntersectionIterator.java:95)
	at org.apache.cassandra.index.sai.iterators.KeyRangeIntersectionIterator.computeNext(KeyRangeIntersectionIterator.java:39)
	at org.apache.cassandra.utils.AbstractGuavaIterator.tryToComputeNext(AbstractGuavaIterator.java:122)
	at org.apache.cassandra.index.sai.iterators.KeyRangeIterator.tryToComputeNext(KeyRangeIterator.java:129)
	at org.apache.cassandra.utils.AbstractGuavaIterator.hasNext(AbstractGuavaIterator.java:116)
	at org.apache.cassandra.index.sai.plan.StorageAttachedIndexSearcher$ResultRetriever.nextKey(StorageAttachedIndexSearcher.java:274)
	at org.apache.cassandra.index.sai.plan.StorageAttachedIndexSearcher$ResultRetriever.nextKeyInRange(StorageAttachedIndexSearcher.java:203)
	at org.apache.cassandra.index.sai.plan.StorageAttachedIndexSearcher$ResultRetriever.nextSelectedKeyInRange(StorageAttachedIndexSearcher.java:234)
	at org.apache.cassandra.index.sai.plan.StorageAttachedIndexSearcher$ResultRetriever.nextRowIterator(StorageAttachedIndexSearcher.java:188)
	at org.apache.cassandra.index.sai.plan.StorageAttachedIndexSearcher$ResultRetriever.computeNext(StorageAttachedIndexSearcher.java:169)
	at org.apache.cassandra.index.sai.plan.StorageAttachedIndexSearcher$ResultRetriever.computeNext(StorageAttachedIndexSearcher.java:111)
	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
	at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:91)
	at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:338)
	at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:201)
	at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:186)
	at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:48)
	at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:346)
	at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:2186)
	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2581)
	at org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163)
	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143)
	at relocated.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829) 

 

Unfortunately, there's no tooling for shrinking around SAI just yet, but I have a programmatic repro using INSERT and DELETE statements. I will do my best to post it asap, but thought this can already be useful for visibility.

Issue Links

• links to: GitHub Pull Request #2943
• links to: GitHub Pull Request #2944