[CASSANDRA-10298] Replaced dead node stayed in gossip forever

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.1.12, 2.2.4, 3.0.0
• Severity: Normal
• Since Version: 2.1.5
• Assignee: Dikang Gu
• Reporter: Dikang Gu

Authors

Dikang Gu

Reviewers

Stefania Alborghetti

• Created: 10/Sep/15 06:58
• Updated: 16/Apr/19 09:30
• Resolved: 04/Nov/15 14:08

Description

The dead node stayed in the nodetool status,

DN 10.210.165.55 379.76 GB 256 ? null

And in the log, it throws NPE when trying to remove it.

2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread Thread[GossipStage:1,5,main]
2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
2015-09-10_06:41:22.92455       at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
2015-09-10_06:41:22.92455       at org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201) 
2015-09-10_06:41:22.92455       at org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
2015-09-10_06:41:22.92455       at org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
2015-09-10_06:41:22.92456       at org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
2015-09-10_06:41:22.92457       at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473) 
2015-09-10_06:41:22.92457       at org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
2015-09-10_06:41:22.92457       at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
2015-09-10_06:41:22.92458       at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
2015-09-10_06:41:22.92458       at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49) 
2015-09-10_06:41:22.92458       at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
2015-09-10_06:41:22.92459       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_45]
2015-09-10_06:41:22.92460       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_45]