[CASSANDRA-12633] Assert Errors occurring with OutboundTcpConnection causes Cassandra to slowly stop functioning.

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Fix Version/s: 2.1.x
• Component/s: Legacy/Streaming and Messaging
• Severity: Normal
• Assignee: Unassigned
• Reporter: Jeff Beck
• Created: 13/Sep/16 13:09
• Updated: 16/Apr/19 09:30
• Resolved: 20/Oct/16 05:30

Description

Assert Errors occurring with OutboundTcpConnection causes Cassandra to slowly stop functioning.

There is a possibility of assert errors in ByteBufferUtil.writeWithShortLength which can cause a stop to processing. There is already good error handling it just needs to catch Throwable instead of Exception at https://github.com/apache/cassandra/blob/c26bd91852cbf19d7dba9f62078f5da31b04dbe0/src/java/org/apache/cassandra/net/OutboundTcpConnection.java#L239

An example of when this happens:

ERROR [MessagingService-Outgoing-/10.0.19.152] 2016-05-03 21:43:54,638 CassandraDaemon.java:227 - Exception in thread Thread[MessagingService-Outgoing-/10.0.19.152,5,main]
java.lang.AssertionError: 103265
        at org.apache.cassandra.utils.ByteBufferUtil.writeWithShortLength(ByteBufferUtil.java:290) ~[apache-cassandra-2.1.12.jar:2.1.12]
   at org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(AbstractCType.java:392) ~[apache-cassandra-2.1.12.jar:2.1.12]
     at org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(AbstractCType.java:381) ~[apache-cassandra-2.1.12.jar:2.1.12]
     at org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.java:271) ~[apache-cassandra-2.1.12.jar:2.1.12]
     at org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.java:259) ~[apache-cassandra-2.1.12.jar:2.1.12]
     at org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQueryFilter.java:503) ~[apache-cassandra-2.1.12.jar:2.1.12]
   at org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQueryFilter.java:490) ~[apache-cassandra-2.1.12.jar:2.1.12]
     at org.apache.cassandra.db.SliceFromReadCommandSerializer.serialize(SliceFromReadCommand.java:168) ~[apache-cassandra-2.1.12.jar:2.1.12]
 at org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:143) ~[apache-cassandra-2.1.12.jar:2.1.12]
   at org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:132) ~[apache-cassandra-2.1.12.jar:2.1.12]
   at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:121) ~[apache-cassandra-2.1.12.jar:2.1.12]
      at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:330) ~[apache-cassandra-2.1.12.jar:2.1.12]
    at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:282) ~[apache-cassandra-2.1.12.jar:2.1.12]
   at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218) ~[apache-cassandra-2.1.12.jar:2.1.12]

The exact error happened when you sent a read request via cql for a clustering column value with more than 64k chars much like CASSANDRA-8280 but on reads. That exact error case is no longer reproducable in the latest versions of C* but in general it seems like it should still handle the possible asserts.

Issue Links

• duplicates: CASSANDRA-11882 Clustering Key with ByteBuffer size > 64k throws Assertion Error
• relates to: CASSANDRA-11882 Clustering Key with ByteBuffer size > 64k throws Assertion Error