[CASSANDRA-16588] NPE getting host_id in Gossiper.isSafeForStartup

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.11.11, 4.0-rc2, 4.0
• Component/s: Cluster/Gossip
• Bug Category: Availability
• Severity: Normal
• Complexity: Normal
• Discovered By: Unit Test
• Platform: All
• Since Version: NA

Source Control Link

https://github.com/apache/cassandra/commit/f6d19512c4d79f800371da1e54dfe01cae5d894e

Test and Documentation Plan

includes test

• Assignee: Brandon Williams
• Reporter: Brandon Williams

Authors

Brandon Williams

Reviewers

Sam Tunnicliffe

• Created: 10/Apr/21 16:17
• Updated: 24/Jun/21 21:19
• Resolved: 21/Apr/21 22:31

Description

As seen here: https://ci-cassandra.apache.org/job/Cassandra-devbranch/604/testReport/junit/org.apache.cassandra.distributed.upgrade/MixedModeGossipTest/testStatusFieldShouldExistInOldVersionNodesEdgeCase/

java.lang.NullPointerException
	at org.apache.cassandra.gms.Gossiper.isSafeForStartup(Gossiper.java:952)
	at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:657)
	at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:933)
	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:784)
	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:729)
	at org.apache.cassandra.distributed.impl.Instance.lambda$startup$10(Instance.java:541)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

I believe what is happening is a GossipDigestAck has been queued to ack the shutdown state from the node on the seed, but isn't actually sent until the node has restarted and gone into shadow. Since the ack contains the node's IP, it assumes a host_id will be there but since this is not an actual shadow response, it is not.

Issue Links

• is a parent of: CASSANDRA-16632 Add gossip tests from CASSANDRA-16588
• is duplicated by: CASSANDRA-14155 [TRUNK] Gossiper somewhat frequently hitting an NPE on node startup with dtests at org.apache.cassandra.gms.Gossiper.isSafeForStartup(Gossiper.java:769)
• links to: CASSANDRA-16588
• links to: CASSANDRA-16588-trunk