[CASSANDRA-13700] Heartbeats can cause gossip information to go permanently missing on certain nodes

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 2.1.19, 2.2.11, 3.0.15, 3.11.1, 4.0-alpha1, 4.0
• Component/s: Legacy/Distributed Metadata
• Bug Category: Availability - Unavailable
• Severity: Critical
• Assignee: Joel Knighton
• Reporter: Joel Knighton

Authors

Joel Knighton

Reviewers

Jason Brown

• Created: 19/Jul/17 21:50
• Updated: 15/May/20 08:01
• Resolved: 24/Jul/17 20:29

Description

In Gossiper.getStateForVersionBiggerThan, we add the HeartBeatState from the corresponding EndpointState to the EndpointState to send. When we're getting state for ourselves, this means that we add a reference to the local HeartBeatState. Then, once we've built a message (in either the Syn or Ack handler), we send it through the MessagingService. In the case that the MessagingService is sufficiently slow, the GossipTask may run before serialization of the Syn or Ack. This means that when the GossipTask acquires the gossip taskLock, it may increment the HeartBeatState version of the local node as stored in the endpoint state map. Then, when we finally serialize the Syn or Ack, we'll follow the reference to the HeartBeatState and serialize it with a higher version than we saw when constructing the Ack or Ack2.

Consider the case where we see HeartBeatState with version 4 when constructing an Ack and send it through the MessagingService. Then, we add some piece of state with version 5 to our local EndpointState. If GossipTask runs and increases the HeartBeatState version to 6 before the MessageOut containing the Ack is serialized, the node receiving the Ack will believe it is current to version 6, despite the fact that it has never received a message containing the ApplicationState tagged with version 5.

I've reproduced in this in several versions; so far, I believe this is possible in all versions.