[CASSANDRA-10596] AssertionError in ReadCommand after upgrade to 3.0

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Not A Problem
• Fix Version/s: 3.0.0
• Severity: Normal
• Assignee: Unassigned
• Reporter: Philip Thompson
• Created: 26/Oct/15 20:09
• Updated: 16/Apr/19 09:30
• Resolved: 26/Oct/15 21:09

Description

The dtest upgrade_through_versions_test.TestUpgrade_from_2_2_latest_tag_to_cassandra_3_0_HEAD.rolling_upgrade_test is failing.

See: http://cassci.datastax.com/view/Upgrades/job/cassandra_upgrade_to_3.0_proto_v4/4/testReport/upgrade_through_versions_test/TestUpgrade_from_2_2_latest_tag_to_cassandra_3_0_HEAD/rolling_upgrade_test_2/

The following exception shows up in the log

Unexpected error in node1 node log: ['ERROR [SharedPool-Worker-2] 2015-08-17 22:30:30,531 Message.java:611 - Unexpected exception during request; channel = [id: 0xee05d108, /127.0.0.1:39640 => /127.0.0.1:9042] java.lang.AssertionError: null \tat org.apache.cassandra.db.ReadCommand$Serializer.serializedSize(ReadCommand.java:520) ~[main/:na] \tat org.apache.cassandra.db.ReadCommand$Serializer.serializedSize(ReadCommand.java:461) ~[main/:na] \tat org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166) ~[main/:na] \tat org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:72) ~[main/:na] \tat org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:583) ~[main/:na] \tat org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:733) ~[main/:na] \tat org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:676) ~[main/:na] \tat org.apache.cassandra.net.MessagingService.sendRRWithFailure(MessagingService.java:659) ~[main/:na] \tat org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:103) ~[main/:na] \tat org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:76) ~[main/:na] \tat org.apache.cassandra.service.AbstractReadExecutor$AlwaysSpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:323) ~[main/:na] \tat org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1599) ~[main/:na] \tat org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1554) ~[main/:na] \tat org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1501) ~[main/:na] \tat org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1420) ~[main/:na] \tat org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:457) ~[main/:na] \tat org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:232) ~[main/:na] \tat org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:202) ~[main/:na] \tat org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:72) ~[main/:na] \tat org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:204) ~[main/:na] \tat org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:470) ~[main/:na] \tat org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:447) ~[main/:na] \tat org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:139) ~[main/:na] \tat org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [main/:na] \tat org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [main/:na] \tat io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] \tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] \tat io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] \tat io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] \tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_51] \tat org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [main/:na] \tat org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [main/:na] \tat java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] ERROR [SharedPool-Worker-3] 2015-08-17 22:30:30,748

I've attached the system.log files from the three ccm nodes in the test.