[CASSANDRA-16518] Node restart during joining sets protocol version to V3

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.27, 3.11.13
• Component/s: Messaging/Client
• Bug Category: Correctness
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Impacts: Clients
• Since Version: 3.0.19

Source Control Link

https://github.com/apache/cassandra/commit/4a15c5ad5a1a6b29be9aac6a8133f4dd936e1379

Test and Documentation Plan

manual test

• Assignee: Stefan Miklosovic
• Reporter: Joseph Clay

Authors

Stefan Miklosovic

Reviewers

Brandon Williams, Sam Tunnicliffe

• Created: 15/Mar/21 02:18
• Updated: 13/Mar/22 19:06
• Resolved: 08/Mar/22 22:44

Description

While joining nodes to a cluster, an old node crashed. The old node was recovered however clients (datastax java) refused to connect to it.

The driver error:

Detected added or restarted Cassandra host /<ip>:<port> but ignoring it since it does not support the version V4 of the native protocol which is currently in use.

In the recovered node cassandra logs:

INFO  o.a.c.transport.ConfiguredLimit Detected peers which do not fully support protocol V4. Capping max negotiable version to V3

I confirmed that ALL the nodes in the cluster, joining or otherwise, were apache-cassandra-3.11.6 so that error message was rather confusing.

Eventually after digging through the code we got to the bottom of the issue:

https://issues.apache.org/jira/browse/CASSANDRA-15193 adds a check for node version, which reverts the protocol version to V3 if any peer fails the version check. Joining nodes have NULL for their version in the peers table, which fails the version check.