[CASSANDRA-11128] Unexpected exception during request; AssertionError: null

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.4, 3.4
• Severity: Normal
• Assignee: Sylvain Lebresne
• Reporter: Russ Hatch

Authors

Sylvain Lebresne

Reviewers

Stefania Alborghetti

• Created: 06/Feb/16 02:13
• Updated: 16/Apr/19 09:30
• Resolved: 05/Apr/16 13:27

Description

Post upgrade from 2.1 to 3.0.3 (3.0.3-tentative) I'm intermittently seeing this exception:

['ERROR [SharedPool-Worker-3] 2016-02-05 22:38:08,444 Message.java:611 - Unexpected exception during request; channel = [id: 0xb5fa21d0, /127.0.0.1:46096 => /127.0.0.1:9042]\njava.lang.AssertionError: null org.apache.cassandra.db.ReadCommand$Serializer.serializedSize(ReadCommand.java:619) ~[main/:na] org.apache.cassandra.db.ReadCommand$Serializer.serializedSize(ReadCommand.java:523) ~[main/:na]
at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166) ~[main/:na]
at org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:72) ~[main/:na]
at org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:609) ~[main/:na]
at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:758) ~[main/:na]
at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:701) ~[main/:na]
at org.apache.cassandra.net.MessagingService.sendRRWithFailure(MessagingService.java:684) ~[main/:na]
at org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:110) ~[main/:na]
at org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85) ~[main/:na]
at org.apache.cassandra.service.AbstractReadExecutor$SpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:264) ~[main/:na]
at org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1689) ~[main/:na]
at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1644) ~[main/:na]
at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1591) ~[main/:na]
at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1510) ~[main/:na]
at org.apache.cassandra.db.SinglePartitionReadCommand.execute(SinglePartitionReadCommand.java:302) ~[main/:na]
at org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:67) ~[main/:na]
at org.apache.cassandra.service.pager.SinglePartitionPager.fetchPage(SinglePartitionPager.java:34) ~[main/:na]
at org.apache.cassandra.cql3.statements.SelectStatement$Pager$NormalPager.fetchPage(SelectStatement.java:302) ~[main/:na]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:338) ~[main/:na]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:214) ~[main/:na]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:76) ~[main/:na]
at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206) ~[main/:na]
at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:472) ~[main/:na]
at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:449) ~[main/:na]
at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130) ~[main/:na]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [main/:na]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [main/:na]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_51]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [main/:na]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [main/:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]']

Issue Links

• breaks: CASSANDRA-13274 Fix code to not exchange schema across major versions
• is related to: CASSANDRA-11393 dtest failure in upgrade_tests.upgrade_through_versions_test.ProtoV3Upgrade_2_1_UpTo_3_0_HEAD.rolling_upgrade_test