[CASSANDRA-14679] Prevent generating new tokens on a node when data exists

• Type: Bug
• Status: Open
• Priority: Urgent
• Resolution: Unresolved
• Component/s: Local/Startup and Shutdown
• Severity: Critical
• Assignee: Unassigned
• Reporter: Michael Semb Wever
• Created: 30/Aug/18 06:08
• Updated: 16/Apr/19 09:29

Description

Data loss is possible if a node starts up without system.local data available.

If a node restarts and its system.local data is unavailable it will generate new tokens. This will cause range movements in the cluster causing potential data loss, as these range movements are not part of a bootstrap/decommission and leaves orphaned data around the cluster.

This can happen if a node restarts without a JBOD entry available, or if the cassandra.yaml changes and leaves a JBOD entry out.

If a node starts up, finds data but not its system.local it should not generate new tokens. Neither should it assign itself a new Host ID.

This is described in more detail in http://thelastpickle.com/blog/2018/08/22/the-fine-print-when-using-multiple-data-directories.html