[CASSANDRA-5780] nodetool status and ring report incorrect/stale information after decommission

• Type: Bug
• Status: Open
• Priority: Low
• Resolution: Unresolved
• Fix Version/s: 2.1.x
• Component/s: Tool/nodetool
• Labels: lhf ponies qa-resolved
• Severity: Low
• Complexity: Low Hanging Fruit
• Assignee: John Sumsion
• Reporter: Peter Haggerty

Authors

John Sumsion

Reviewers

T Jake Luciani

• Tester: Ryan McGuire
• Created: 19/Jul/13 21:10
• Updated: 13/Jun/19 18:28

Description

Cassandra 1.2.6 ring of 12 instances, each with 256 tokens.

Decommission 3 of the 12 nodes, one after another resulting a 9 instance ring.

The 9 instances of cassandra that are in the ring all correctly report nodetool status information for the ring and have the same data.

After the first node is decommissioned:
"nodetool status" on "decommissioned-1st" reports 11 nodes

After the second node is decommissioned:
"nodetool status" on "decommissioned-1st" reports 11 nodes
"nodetool status" on "decommissioned-2nd" reports 10 nodes

After the second node is decommissioned:
"nodetool status" on "decommissioned-1st" reports 11 nodes
"nodetool status" on "decommissioned-2nd" reports 10 nodes
"nodetool status" on "decommissioned-3rd" reports 9 nodes

The storage load information is similarly stale on the various decommissioned nodes. The nodetool status and ring commands continue to return information as if they were part of a cluster and they appear to return the last information that they saw.

In contrast the nodetool info command fails with an exception, which isn't ideal but at least indicates that there was a failure rather than returning stale information.

Issue Links

• relates to: CASSANDRA-8801 Decommissioned nodes are willing to rejoin the cluster if restarted