[CASSANDRA-18591] assertion failure when reading after writes and column drop are executed in parallel

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Component/s: Local/Other
• Bug Category: Correctness - Recoverable Corruption / Loss
• Severity: Normal
• Complexity: Normal
• Discovered By: Adhoc Test
• Platform: All
• Assignee: Jakub Zytka
• Reporter: Jakub Zytka

Authors

Jakub Zytka

• Created: 13/Jun/23 11:09
• Updated: 26/Jun/23 16:24

Description

This is another face of the problem reported in CASSANDRA-18589, but this problem may happen regardless of whether the dropped column is complex.

 

When writing data in parallel with dropping a column, the subsequent reads may fail with AssertionError:  until the affected sstable is compacted. 

ERROR [node1_isolatedExecutor:3] node1 2023-06-12 15:45:28,058 Exception in thread Thread[node1_ReadStage-2,10,SharedPool]
java.lang.RuntimeException: java.lang.AssertionError: Caught an error while trying to process the command: SELECT * FROM ks.tab LIMIT 500
0 ALLOW FILTERING
        at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:108)
        at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:45)
        at org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:430)
        at org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.AssertionError: Caught an error while trying to process the command: SELECT * FROM ks.tab LIMIT 5000 ALLOW FILTERING
        at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:83)
        at org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:78)
        at org.apache.cassandra.net.InboundSink$Filtered.accept(InboundSink.java:64)
        at org.apache.cassandra.net.InboundSink$Filtered.accept(InboundSink.java:50)
        at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:97)
        ... 6 common frames omitted
Caused by: java.lang.AssertionError: v
        at org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$serializeRowBody$0(UnfilteredSerializer.java:255)
        at org.apache.cassandra.utils.btree.BTree.applyValue(BTree.java:1762)
        at org.apache.cassandra.utils.btree.BTree.applyLeaf(BTree.java:1770)
        at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1785)
        at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1811)
        at org.apache.cassandra.db.rows.BTreeRow.apply(BTreeRow.java:203)
        at org.apache.cassandra.db.rows.UnfilteredSerializer.serializeRowBody(UnfilteredSerializer.java:246)
        at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:212)
        at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:141)
        at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:129)
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:153)
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:102)
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87)
        at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:308)
        at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:201)
        at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:186)
        at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:48)
        at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:338)
        at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:64)
        ... 10 common frames omitted

It is likely that the intended fix of CASSANDRA-18589 will make this issue go away.

For now, I'm reporting two different tickets to not mix two discussions together, should they happen.