[CASSANDRA-16525] Gossip STATUS can be either missing during upgrade or stale after upgrade

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-rc1, 4.0
• Component/s: Cluster/Gossip
• Bug Category: Correctness - Recoverable Corruption / Loss
• Severity: Normal
• Complexity: Normal
• Discovered By: Unit Test
• Platform: All
• Since Version: 4.0-alpha1

Source Control Link

https://github.com/apache/cassandra/commit/c591978f4d265e42d0132418005ba63a99278c75 https://github.com/apache/cassandra/commit/1ca5769c9adfd2e8bad3b7eea83112ad14cb05b2

Test and Documentation Plan

jvm upgrade dtest

• Assignee: Yifan Cai
• Reporter: Yifan Cai

Authors

Yifan Cai

Reviewers

Michael Semb Wever

• Created: 16/Mar/21 22:56
• Updated: 16/Mar/22 15:36
• Resolved: 19/Mar/21 15:57

Description

In 4.0, new application states are added in Gossip and the corresponding old ones are deprecated, e.g. STATUS and the successor STATUS_WITH_PORT.

There are 2 issues discovered by the jvm (upgrade) dtest. First, the STATUS field of a peer in the lower version (e.g. 3.0) node can be missing. Second, it is possible the STATUS coexist with the new state STATUS_WITH_PORT in the 4.0 nodes after cluster is fully upgraded and the STATUS field can becomes stale as the 4.0 node filters out when applying new state.

The first issue can happen in this scenario. During upgrade, node1 and node2 are in v4, and node3 is still in v3. If node3 only gets the gossip info regarding node2 from node1, the STATUS field of node2 will be missing in node3's local state, which is unexpected. There could be many reasons that node3 does not exchange gossip with node2 directly, e.g. network issue between node2 and node3, or node2 simply does not select node3 when initiating the gossip round. Gossip should be resilient to it. I have a jvm upgrade dtest to demonstrate the unexpected behavior.

The cause of the second issue is more subtle. Heartbeat update happens as part of the Gossip task and outside of the GossipStage. When node2 just update its local application state and received a SYN from node1, node 2 just replies its gossip state without updating the heartbeat version. When node1 receives it, it first filters out the legacy STATUS field, and only saves the new one. So far so good. However, node2 soon updates its heart beat, and node1 realizes that its local version is less than the remove (node2) version in the next gossip round. So node2 sends STATUS along to node1. Because it does not come together with the new field, node1 does not filter it out when receiving. Boo! Node1 now has the STATUS field from node2. Such field can become stale and diverge with its successor in a live cluster. The jvm upgrade test testStatusFieldShouldExistInOldVersionNodes can fairly easy to reproduce it when the entire cluster is upgraded. And there is another jvm dtest (with source changes to help make deterministic result, see the attached Demonstrate-a-scenario-that-a-node-may-hold-the-stale-status.patch) that demonstrates the STATUS can be replicated to the peer and become stale.

The fix is to
1) retain the legacy fields if the cluster is still in mixed mode
2) remove the legacy field when cluster is fully upgraded

Issue Links

• is caused by: CASSANDRA-16381 nodetool removenode error “Conflicting replica added”