[CASSANDRA-8801] Decommissioned nodes are willing to rejoin the cluster if restarted

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0 alpha 1
• Severity: Normal
• Assignee: Brandon Williams
• Reporter: Eric Stevens

Authors

Brandon Williams

Reviewers

Carl Yeksigian

• Tester: Jim Witschey
• Created: 13/Feb/15 15:03
• Updated: 16/Apr/19 09:31
• Resolved: 19/May/15 15:47

Description

This issue comes from the Cassandra user group.

If a node which was successfully decommissioned gets restarted with its data directory in tact, it will rejoin the cluster immediately going to UN and beginning to serve client requests.

This is wrong - the node has consistency issues, having missed any writes while it was offline because no hinted handoffs were being kept. And in the best case scenario (it's spotted and remediated immediately), near-100% overstreaming will still occur.

Also, whatever reasons the operator had for decommissioning the node would presumably still be valid, so this action may threaten cluster stability if the node is underpowered or suffering hardware issues.

But what elevates this to critical is that if the node had been offline longer than gc_grace_seconds, it may cause permanent and unrecoverable consistency issues due to data resurrection.

Recommendation:

A node should remember that it was decommissioned and refuse to rejoin a cluster without at least a -Dflag forcing it to.

Issue Links

• is related to: CASSANDRA-5780 nodetool status and ring report incorrect/stale information after decommission
• links to: dtest