[CASSANDRA-16313] Connection failure to down node spams debug.log

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 3.0.x, 3.11.x
• Component/s: Cluster/Gossip
• Bug Category: Degradation - Other Exception
• Severity: Low
• Complexity: Low Hanging Fruit
• Discovered By: User Report
• Platform: All
• Assignee: Krishna Vadali
• Reporter: Paulo Motta

Authors

Krishna Vadali

• Created: 04/Dec/20 14:30
• Updated: 04/Dec/20 15:39

Description

When a node becomes down a periodic attempt is made to try to reconnect to the node. However, the connection error spams the debug.log of all nodes in the cluster with the following message:

DEBUG [MessagingService-Outgoing-/XX.YY.ZZ.WW-Gossip] 2020-12-04 09:13:58,999 OutboundTcpConnection.java:546 - Unable to connect to /XX.YY.ZZ.WW
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_172]
	at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_172]
	at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_172]
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[na:1.8.0_172]
	at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:146) ~[apache-cassandra-3.11.3.jar:3.11.3]
	at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:132) ~[apache-cassandra-3.11.3.jar:3.11.3]
	at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:434) [apache-cassandra-3.11.3.jar:3.11.3]
	at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:262) [apache-cassandra-3.11.3.jar:3.11.3]

I've seen cases where this rolled the entire debug.log which can difficult troubleshooting.

We should probably guard this in a NoSpamLogger.