[CASSANDRA-15120] Nodes that join the ring while another node is MOVING build an invalid view of the token ring

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.19, 3.11.5, 4.0-alpha1, 4.0
• Component/s: Cluster/Gossip, Cluster/Membership
• Bug Category: Correctness - Consistency
• Severity: Critical
• Complexity: Challenging
• Discovered By: User Report
• Platform: All
• Since Version: 3.0.0

Source Control Link

https://github.com/apache/cassandra/commit/e4b5d9818f003be2b9091c48f8435d29202ffe2d

Test and Documentation Plan

regression test included

• Assignee: Benedict Elliott Smith
• Reporter: Benedict Elliott Smith

Authors

Benedict Elliott Smith

Reviewers

Sam Tunnicliffe

• Created: 07/May/19 14:32
• Updated: 15/May/20 08:00
• Resolved: 03/Jun/19 15:07

Description

Gossip only updates the token metadata for nodes in the NORMAL, SHUTDOWN or LEAVING* statuses.  MOVING and REMOVING_TOKEN nodes do not have their ring information updated (nor do others, but these other states should only be taken by nodes that are not members of the ring).  

If a node missed the most recent token-modifying events because they were not a member of the ring when they happened (or because Gossip was delayed to them), they will retain an invalid view of the ring until the node enters the one of the NORMAL, SHUTDOWN or LEAVING states.

*LEAVING is populated differently, however, and in a probably unsafe manner that this work will also address.