[CASSANDRA-18543] Waiting for gossip to settle does not wait for live endpoints

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.11.16, 4.0.11, 4.1.3, 5.0-alpha1, 5.0
• Component/s: Cluster/Gossip
• Bug Category: Availability
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: NA

Source Control Link

https://github.com/apache/cassandra/commit/64d953fdcac91715898454a3cb04d932910a70b7

Test and Documentation Plan

CI

• Assignee: Stefan Miklosovic
• Reporter: Cameron Zemek

Authors

Stefan Miklosovic

Reviewers

Brandon Williams

• Created: 23/May/23 00:36
• Updated: 09/Feb/24 19:58
• Resolved: 15/Jun/23 13:58

Description

When a node starts it will get endpoint states (via shadow round) but have all nodes marked as down. The problem is the wait to settle only checks the size of endpoint states is stable before starting Native transport. Once native transport starts it will receive queries and fail consistency levels such as LOCAL_QUORUM since it still thinks nodes are down.

This is problem for a number of large clusters for our customers. The cluster has quorum but due to this issue a node restart is causing a bunch of query errors.

My initial solution to this was to only check live endpoints size in addition to size of endpoint states. This worked but I noticed in testing this fix that there also a lot of duplication of checking the same node (via Echo messages) for liveness. So the patch also removes this duplication of checking node is UP in markAlive.

The final problem I found while testing is sometimes could still not see a change in live endpoints due to only 1 second polling, so the patch allows for overridding the settle parameters. I could not reliability reproduce this but think its worth providing a way to override these hardcoded values.

Issue Links

• links to: GitHub Pull Request #2361
• links to: GitHub Pull Request #2376