[CASSANDRA-17252] ConnectionLimitHandler may leaks connection count if remote connection drops

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.27, 3.11.13, 4.0.4, 4.1-alpha1, 4.1
• Component/s: Messaging/Client
• Bug Category: Degradation - Resource Management
• Severity: Low
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 2.0.15

Source Control Link

https://github.com/apache/cassandra/commit/ad532ef1ddd07eaa3cea658b7b3bbdf13bd1bdb4

Test and Documentation Plan

Rely on existing client tests.  The exception is hard to reliably provoke - I ended up running many iterations on my laptop with breakpoints set to hit the null while looping connections.

• Assignee: Jon Meredith
• Reporter: Jon Meredith

Authors

Jon Meredith

Reviewers

Caleb Rackliffe

• Created: 11/Jan/22 00:10
• Updated: 27/May/22 19:24
• Resolved: 19/Mar/22 14:17

Description

In some cases, Netty does not return the original IP used for per-IP counting when the channel becomes inactive,
which throws an NPE before decrementing the active per-IP count.

java.lang.NullPointerException
at org.apache.cassandra.transport.ConnectionLimitHandler.channelInactive(ConnectionLimitHandler.java:101)
       at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
       at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
       at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
       at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389)
       at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)
       at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
       at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
       at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
       at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
       at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
       at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
       at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
       at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819)
       at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
       at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
       at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
       at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
       at java.base/java.lang.Thread.run(Thread.java:834)