[CASSANDRA-16097] DigestResolver.getData throws AssertionError since dataResponse is null

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-beta4, 4.0
• Component/s: Consistency/Coordination
• Bug Category: Availability - Response Crash
• Severity: Normal
• Complexity: Normal
• Discovered By: Performance Regression Test
• Platform: All
• Since Version: 4.0-alpha1

Source Control Link

https://github.com/apache/cassandra/commit/4c103447af3c4829e3a1c733bed3952fd059af08

Test and Documentation Plan

Testing:
I had a manual test (linked in comments) that reproduced this with 100% certainty over time on my platform, but it was not deterministic. I'm not sure if it's worth converting into a dtest because it's based on timing for both the local error, and spec exec decisions are not deterministic.

No docs

• Assignee: Adam Holmberg
• Reporter: David Capwell

Authors

Adam Holmberg, Caleb Rackliffe

Reviewers

Berenguer Blasi, Brandon Williams, Caleb Rackliffe, David Capwell

• Created: 02/Sep/20 23:48
• Updated: 03/Jan/21 16:58
• Resolved: 08/Dec/20 19:04

Description

Was running a benchmark at LOCAL_ONE and eventually saw the below exception

2020-09-02 21:08:59,872 ERROR [Native-Transport-Requests-35] org.apache.cassandra.transport.Message - Unexpected exception during request; channel = [id: 0x13bb89d4, L:/10.14.92.74:9042 - R:/10.14.89.248:47112]
java.lang.AssertionError
       at org.apache.cassandra.service.reads.DigestResolver.getData(DigestResolver.java:77) ~[apache-cassandra-4.0.0-beta3.jar:4.0.0-beta3]
       at org.apache.cassandra.service.reads.AbstractReadExecutor.awaitResponses(AbstractReadExecutor.java:390) ~[apache-cassandra-4.0.0-beta3.jar:4.0.0-beta3]
       at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1821) ~[apache-cassandra-4.0.0-beta3.jar:4.0.0-beta3]
       at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1711) ~[apache-cassandra-4.0.0-beta3.jar:4.0.0-beta3]
       at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1628) ~[apache-cassandra-4.0.0-beta3.jar:4.0.0-beta3]
       at org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1097) ~[apache-cassandra-4.0.0-beta3.jar:4.0.0-beta3]
       at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:294) ~[apache-cassandra-4.0.0-beta3.jar:4.0.0-beta3]
       at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:246) ~[apache-cassandra-4.0.0-beta3.jar:4.0.0-beta3]
       at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:88) ~[apache-cassandra-4.0.0-beta3.jar:4.0.0-beta3]
       at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:216) ~[apache-cassandra-4.0.0-beta3.jar:4.0.0-beta3]
       at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:498) ~[apache-cassandra-4.0.0-beta3.jar:4.0.0-beta3]
       at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:476) ~[apache-cassandra-4.0.0-beta3.jar:4.0.0-beta3]
       at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:138) ~[apache-cassandra-4.0.0-beta3.jar:4.0.0-beta3]
       at org.apache.cassandra.transport.Message$Request.execute(Message.java:253) ~[apache-cassandra-4.0.0-beta3.jar:4.0.0-beta3]
       at org.apache.cassandra.transport.Message$Dispatcher.processRequest(Message.java:725) ~[apache-cassandra-4.0.0-beta3.jar:4.0.0-beta3]
       at org.apache.cassandra.transport.Message$Dispatcher.lambda$channelRead0$0(Message.java:630) ~[apache-cassandra-4.0.0-beta3.jar:4.0.0-beta3]
       at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
       at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-4.0.0-beta3.jar:4.0.0-beta3]
       at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119) [apache-cassandra-4.0.0-beta3.jar:4.0.0-beta3]
       at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.50.Final.jar:4.1.50.Final]
       at java.base/java.lang.Thread.run(Thread.java:834) [?:?]

This exception was not frequent, out of the whole run (3h) only saw this twice.

Issue Links

• relates to: CASSANDRA-16103 Invalid serialized size for responses
• relates to: CASSANDRA-16106 BufferOverflow exception while writing response to buffer