[CASSANDRA-11393] dtest failure in upgrade_tests.upgrade_through_versions_test.ProtoV3Upgrade_2_1_UpTo_3_0_HEAD.rolling_upgrade_test

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.9, 3.8
• Component/s: Legacy/Coordination, Legacy/Streaming and Messaging
• Labels: dtest
• Severity: Normal
• Assignee: Benjamin Lerer
• Reporter: Philip Thompson

Authors

Benjamin Lerer

Reviewers

Tom Hobbs

• Created: 21/Mar/16 18:04
• Updated: 16/Apr/19 09:30
• Resolved: 14/Jul/16 09:56

Description

We are seeing a failure in the upgrade tests that go from 2.1 to 3.0

node2: ERROR [SharedPool-Worker-2] 2016-03-10 20:05:17,865 Message.java:611 - Unexpected exception during request; channel = [id: 0xeb79b477, /127.0.0.1:39613 => /127.0.0.2:9042]
java.lang.AssertionError: null
	at org.apache.cassandra.db.ReadCommand$LegacyReadCommandSerializer.serializedSize(ReadCommand.java:1208) ~[main/:na]
	at org.apache.cassandra.db.ReadCommand$LegacyReadCommandSerializer.serializedSize(ReadCommand.java:1155) ~[main/:na]
	at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166) ~[main/:na]
	at org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:72) ~[main/:na]
	at org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:609) ~[main/:na]
	at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:758) ~[main/:na]
	at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:701) ~[main/:na]
	at org.apache.cassandra.net.MessagingService.sendRRWithFailure(MessagingService.java:684) ~[main/:na]
	at org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:110) ~[main/:na]
	at org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85) ~[main/:na]
	at org.apache.cassandra.service.AbstractReadExecutor$AlwaysSpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:330) ~[main/:na]
	at org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1699) ~[main/:na]
	at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1654) ~[main/:na]
	at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1601) ~[main/:na]
	at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1520) ~[main/:na]
	at org.apache.cassandra.db.SinglePartitionReadCommand.execute(SinglePartitionReadCommand.java:302) ~[main/:na]
	at org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:67) ~[main/:na]
	at org.apache.cassandra.service.pager.SinglePartitionPager.fetchPage(SinglePartitionPager.java:34) ~[main/:na]
	at org.apache.cassandra.cql3.statements.SelectStatement$Pager$NormalPager.fetchPage(SelectStatement.java:297) ~[main/:na]
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:333) ~[main/:na]
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:209) ~[main/:na]
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:76) ~[main/:na]
	at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206) ~[main/:na]
	at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:472) ~[main/:na]
	at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:449) ~[main/:na]
	at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130) ~[main/:na]
	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [main/:na]
	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [main/:na]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]
	at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_51]
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [main/:na]
	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [main/:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]

example failure:

http://cassci.datastax.com/job/upgrade_tests-all/24/testReport/upgrade_tests.upgrade_through_versions_test/ProtoV3Upgrade_2_1_UpTo_3_0_HEAD/rolling_upgrade_test

Failed on CassCI build upgrade_tests-all #24

The stack trace and context match that of CASSANDRA-10122. It looks like it may be the same issue.

Issue Links

• is related to: CASSANDRA-11767 dtest failure in upgrade_tests.upgrade_through_versions_test.ProtoV3Upgrade_AllVersions_EndsAt_Trunk_HEAD.rolling_upgrade_test
• relates to: CASSANDRA-11128 Unexpected exception during request; AssertionError: null