[CASSANDRA-13323] IncomingTcpConnection closed due to one bad message

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Fix Version/s: 3.0.15
• Severity: Normal
• Assignee: Simon Zhou
• Reporter: Simon Zhou

Authors

Simon Zhou

• Created: 10/Mar/17 23:46
• Updated: 01/Aug/21 12:29
• Resolved: 16/Aug/17 23:56

Description

We got this exception:

WARN  [MessagingService-Incoming-/****] 2017-02-14 17:33:33,177 IncomingTcpConnection.java:101 - UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for cfId 2a3ab630-df74-11e6-9f81-b56251e1559e. If a table was just created, this is likely due to the schema not being fully propagated.  Please wait for schema agreement on table creation.
    at org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1336) ~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:660) ~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:635) ~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:131) ~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:113) ~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) ~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201) ~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) ~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) ~[apache-cassandra-3.0.10.jar:3.0.10]

Also we saw this log in another host indicating it needs to re-connect:

INFO  [HANDSHAKE-/****] 2017-02-21 13:37:50,216 OutboundTcpConnection.java:515 - Handshaking version with /****

The reason is that the node was receiving hinted data for a dropped table. This may happen with other messages as well. On Cassandra side, IncomingTcpConnection shouldn't close on just one bad message, even though it will be restarted soon later by SocketThread in MessagingService.

Issue Links

• duplicates: CASSANDRA-9289 Recover from unknown table when deserializing internode messages
• is duplicated by: CASSANDRA-9289 Recover from unknown table when deserializing internode messages