[CASSANDRA-16156] Decomissioned nodes are picked for gossip when unreachable nodes are considered for gossiping

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.23, 3.11.9, 4.0
• Component/s: Cluster/Gossip
• Bug Category: Code - Bug - Unclear Impact
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 3.0.21

Source Control Link

https://github.com/apache/cassandra/commit/bf1ea945b6fb1efb1730a3b8973325a26d51be39

Test and Documentation Plan

Test included into CASSANDRA-15935, can find it here.

• Assignee: Alex Petrov
• Reporter: Alex Petrov

Authors

Alex Petrov

Reviewers

Marcus Eriksson

• Created: 01/Oct/20 14:27
• Updated: 22/Oct/20 07:53
• Resolved: 22/Oct/20 07:53

Description

After node is decommissioned, it is still considered for gossip via “unreachable” nodes, which results into following exceptions:
 

INFO  [node4_Messaging-EventLoop-3-3] node4 2020-09-29 16:37:37,527 NoSpamLogger.java:91 - /127.0.0.4:7012->/127.0.0.1:7012-URGENT_MESSAGES-[no-channel] failed to connect
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:7012
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
 

Trace of the method that attempts to establish connection:

 
org.apache.cassandra.net.MessagingService.getOutbound(MessagingService.java:492)
	at org.apache.cassandra.net.MessagingService.doSend(MessagingService.java:335)
	at org.apache.cassandra.net.OutboundSink$Filtered.accept(OutboundSink.java:55)
	at org.apache.cassandra.net.OutboundSink.accept(OutboundSink.java:70)
	at org.apache.cassandra.net.MessagingService.send(MessagingService.java:327)
	at org.apache.cassandra.net.MessagingService.send(MessagingService.java:314)
	at org.apache.cassandra.gms.Gossiper.sendGossip(Gossiper.java:813)
	at org.apache.cassandra.gms.Gossiper.maybeGossipToUnreachableMember(Gossiper.java:840)
	at org.apache.cassandra.gms.Gossiper.access$400(Gossiper.java:86)
 

LEFT and other nodes that are considered dead should not be picked for gossip with unreachable nodes.

Issue Links

• requires: CASSANDRA-15935 Improve machinery for testing consistency in presence of range movements