[CASSANDRA-14692] join_ring=false populates wrong value into StorageServiceMB and prevents join by nodetool

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 3.11.x
• Component/s: Local/Startup and Shutdown
• Severity: Normal
• Assignee: Unassigned
• Reporter: Roland Johann
• Created: 05/Sep/18 15:35
• Updated: 08/Jul/19 15:51

Description

Restarting a cassandra cluster member with option -Dcassandra.join_ring=false populates wrong value to its StorageServiceMB field Joined which causes the actual trigger to join via nodetool join to abort due to check if Join in StorageServiceMB is true. Via jconsole it's possible as there is no check.

https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/tools/nodetool/Join.java

nodetool status also shows that the node is up and in normal node, on the rest of the cluster node status is  DN. 

nodetool gossipinfo states that the non joined node is in gossip state hibernate.

Came across this issue while evaluated the problem of zombies to integrate into automation processes and the documentation states

To avoid this problem, run nodetool repair on any restored node before rejoining it to its cluster.