[CASSANDRA-11299] AssertionError when querying by secondary index

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Component/s: Feature/2i Index

Environment

Cassandra 3.3

• Severity: Normal
• Assignee: Andres de la Peña
• Reporter: Michał Matłoka

Authors

Andres de la Peña

• Created: 02/Mar/16 23:53
• Updated: 16/Apr/19 09:30

Description

Hi,
Recently we have upgraded from Cassandra 2.2.4 to 3.3. I have issues with one table. When I try to query using any secondary index I get e.g. in cqlsh

Traceback (most recent call last):
  File "/usr/bin/cqlsh.py", line 1249, in perform_simple_statement
    result = future.result()
  File "/usr/share/cassandra/lib/cassandra-driver-internal-only-3.0.0-6af642d.zip/cassandra-driver-3.0.0-6af642d/cassandra/cluster.py", line 3122, in result
    raise self._final_exception
ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation failed - received 0 responses and 1 failures" info={'failures': 1, 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}

Node logs shows then:

[[AWARN  [SharedPool-Worker-2] 2016-03-03 00:47:01,679 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-2,5,main]: {}
java.lang.AssertionError: null
        at org.apache.cassandra.index.internal.composites.CompositesSearcher$1Transform.findEntry(CompositesSearcher.java:225) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.index.internal.composites.CompositesSearcher$1Transform.applyToRow(CompositesSearcher.java:215) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:116) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:133) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:127) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:123) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:292) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1789) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2457) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_66]
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.3.0.jar:3.3.0]
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.3.0.jar:3.3.0]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]

SStables are upgraded, I have tried repair and scrub. I have tried to rebuild indexes, and even remove them and re-add them.It occurs on every cluster node.

Additionally I had seen in this table case where PRIMARY KEY was duplicated!!! (there were two rows with same primary key, by seeing what columns were set I can say one was older, and second was from newer query which sets only a subset of columns)

Issue Links

• is related to: CASSANDRA-11887 Duplicate rows after a 2.2.5 to 3.0.4 migration