[AMQ-6976] GC goes crazy under load. Heap is filled up with TransportConnectionState objects

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Workaround
• Affects Version/s: 5.15.3
• Fix Version/s: 5.15.3
• Component/s: Broker

Environment

Windows Server 2012 R2 - 64bit
230 GB RAM
SSD HD

• Assignee: Unassigned
• Reporter: Markus L.
• Created: 28/May/18 12:06
• Updated: 05/Jan/20 05:41
• Resolved: 16/Jul/18 12:28

Description

We are running a so called "Shared Nothing" setup with several Wildfly 8.2.1 instances (10) on several host machines (4)- all connected to one standalone Activemq installation - Normally we have  600-700 connections in total to our broker.

The Activemq installation is configured to use 8GB RAM.

<broker xmlns="http://activemq.apache.org/schema/core" brokerName="dcdng" id="dcdng" useJmx="true" useShutdownHook="false"
schedulerSupport="false" dataDirectory="C:/activemq/data" advisorySupport="false">

<managementContext>
<managementContext createConnector="true" connectorPort="61610" suppressMBean="endpoint=dynamicProducer,endpoint=Consumer,connectionName=*,destinationName=ActiveMQ.Advisory.*"/>
</managementContext>
 <!-- ... -->
</broker>

The memory usage is set to 500mb which is sufficient since our messages are relative small. (MaxMessageSize: 41442)

 

The system performs great until we put it under high load. Then the heap usage grows and it seems the memory cannot get cleaned up properly. 

At the end I took a heap dump and the MAT prints the following:

So why are there so many TransportConnectionState objects left an not cleaned up properly. When the system performs under "normal" load, everything seems to be fine.