[ARTEMIS-4797] Failover connection references are not always cleaned up in NettyAcceptor, leaking memory

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: OpenWire
• Assignee: Unassigned
• Reporter: Josh Byster
• Created: 05/Jun/24 14:59
• Updated: 24/Oct/24 18:59
• Resolved: 24/Oct/24 18:59

Description

I'm still trying to parse through exactly what conditions this occurs in, since I'm able to reproduce it in a very specific production setup but not in an isolated environment locally.

For context, we have custom slow consumer detection that closes connection IDs with slow consumers. These connections are connected via failover transport using client ActiveMQ Classic 5.16.4 (OpenWire). This seems to be specific to Netty.

It appears this specific order of events causes the connection to not get cleaned up and retained indefinitely on the broker. With frequent kicking of connections, this ends up causing the broker to eventually OOM.

1. Connection is created, ActiveMQServerChannelHandler is created as well
2. ActiveMQServerChannelHandler#createConnection is called, active flag is set true.
3. A few minutes go by, then we call ActiveMQServerControl#closeConnectionWithID with the connection ID.
4. ActiveMQChannelHandler#exceptionCaught gets called—this is the key point that causes issues. The connection is cleaned up if and only if this is not called. The root cause of the exception is AbstractChannel.close(ChannelPromise), however the comment above it says this is normal for failover.
5. The active flag is set to false.
6. ActiveMQChannelHandler#channelInactive gets called, but does not call listener.connectionDestroyed since the active flag is false.
7. The connection is never removed from the connections map in NettyAcceptor, causing a leak and eventual OOM of the broker if it happens frequently enough.

Issue Links

• duplicates: ARTEMIS-4731 Openwire failure detection leak with Windows Clients / Server won't cleanup openwork clients with Exception
• links to: GitHub Pull Request #4960