[CASSANDRA-18968] StartupClusterConnectivityChecker fails on upgrade from 3.X

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0.12, 4.1.4
• Component/s: Local/Startup and Shutdown
• Labels: lhf
• Bug Category: Correctness - API / Semantic Implementation
• Severity: Low
• Complexity: Low Hanging Fruit
• Discovered By: User Report
• Platform: All
• Since Version: 4.0-alpha1

Source Control Link

https://github.com/apache/cassandra/commit/26a7d574f00dfd00cb76363ce46ab544a0b50c3b

Test and Documentation Plan

In this scenario you should see:
WARN [main] 2023-10-27 15:58:22,234 StartupClusterConnectivityChecker.java:183 - Timed out after 10002 milliseconds, was waiting for remaining peers to connect:

{dc1=[X.Y.Z.W, A.B.C.D]}

.

After applying this patch, instead you should see:
Skipping startup connectivity check as some nodes may be running Cassandra version 3 or older which does not support connectivity checking.

• Assignee: Isaac Reath
• Reporter: Paulo Motta

Authors

Isaac Reath

Reviewers

Paulo Motta, Stefan Miklosovic

• Created: 27/Oct/23 16:46
• Updated: 10/Nov/23 10:36
• Resolved: 10/Nov/23 10:35

Description

Starting up a new 4.X node on a 3.x cluster throws the following warning:

WARN  [main] 2023-10-27 15:58:22,234 StartupClusterConnectivityChecker.java:183 - Timed out after 10002 milliseconds, was waiting for remaining peers to connect: {dc1=[X.Y.Z.W, A.B.C.D]}

I think this is because the PING messages used by the startup check are not available on 3.X.

To provide a smoother upgrade experience we should probably disable this check on a mixed version clusters, or skip peers on versions < 4.x when doing the connectivity check.

Issue Links

• relates to: CASSANDRA-14297 Startup checker should wait for count rather than percentage
• relates to: CASSANDRA-13993 Add optional startup delay to wait until peers are ready
• relates to: CASSANDRA-14447 Cleanup StartupClusterConnectivityChecker and PING Verb
• Testing discovered: CASSANDRA-18999 Gossiper::hasMajorVersion3Nodes returns true when a cluster is upgrading patch version without Cassandra 3 nodes.
• links to: GitHub Pull Request #2862
• links to: GitHub Pull Request #2863