[AMQ-7366] ActiveMQ broker always hangs up at intervals in the case of high concurrent connections

• Type: Bug
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 5.15.0
• Fix Version/s: 5.15.0
• Component/s: Broker, Connector, MQTT
• Labels: performance

Environment

activeMQ version:5.15.0

linux OS version：Red Hat Enterprise Linux Server release 6.9 (Santiago)

linux server configuration: Memery:32G、CPU:16、disk:200G

 

• Flags: Important
• Assignee: Unassigned
• Reporter: zhiwei wang
• Created: 17/Dec/19 09:30
• Updated: 13/May/21 20:40

Description

My broker JVM configuration is "- xms16g - xmx16g - xss256k - XX: metaspacesize = 128M".

The transport connector in activemq.xml is configured as follows:

<transportConnector name="sslmqtt" uri="mqtt+ssl://0.0.0.0:1883?maximumConnections=200000&wireFormat.maxFrameSize=104857600&jms.watchTopicAdvisories=false&transport.defaultKeepAlive=30000"/>

At present, it can support 15000 client connections. But every two days or less, ActiveMQ broker will hang up. When I check the logs of ActiveMQ broker, I find the following errors:

1.

org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>60000) long: tcp://180.162.0.129:50014org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>60000) long: tcp://180.162.0.129:50014 at org.apache.activemq.transport.mqtt.MQTTInactivityMonitor$1$1.run(MQTTInactivityMonitor.java:82)[activemq-mqtt-5.15.0.jar:5.15.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_151] at java.lang.Thread.run(Thread.java:748)[:1.8.0_151]2019-12-17 13:48:28,123 | DEBUG | No CONNECT frame received in time for tcp:///101.80.38.31:4751@1883! Throwing InactivityIOException. | org.apache.activemq.transport.mqtt.MQTTInactivityMonitor | InactivityMonitor ReadCheck2019-12-17 13:48:28,124 | DEBUG | Transport Connection to: tcp://101.88.157.200:6892 failed: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>60000) long: tcp://101.88.157.200:6892 | org.apache.activemq.broker.TransportConnection.Transport | MQTTInactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@ddca0bb[State = -1, empty queue]org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>60000) long: tcp://101.88.157.200:6892 at org.apache.activemq.transport.mqtt.MQTTInactivityMonitor$1$1.run(MQTTInactivityMonitor.java:82)[activemq-mqtt-5.15.0.jar:5.15.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_151] at java.lang.Thread.run(Thread.java:748)[:1.8.0_151]2019-12-17 13:48:28,136 | DEBUG | No CONNECT frame received in time for tcp:///114.83.88.242:36665@1883! Throwing InactivityIOException. | org.apache.activemq.transport.mqtt.MQTTInactivityMonitor | InactivityMonitor ReadCheck

2.

javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)[:1.8.0_151] at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)[:1.8.0_151] at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:938)[:1.8.0_151] at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)[:1.8.0_151] at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)[activemq-client-5.15.0.jar:5.15.0] at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634)[activemq-client-5.15.0.jar:5.15.0] at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59)[activemq-client-5.15.0.jar:5.15.0] at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619)[activemq-client-5.15.0.jar:5.15.0] at java.io.DataInputStream.readByte(DataInputStream.java:265)[:1.8.0_151] at org.apache.activemq.transport.mqtt.MQTTWireFormat.unmarshal(MQTTWireFormat.java:86)[activemq-mqtt-5.15.0.jar:5.15.0] at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)[activemq-client-5.15.0.jar:5.15.0] at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)[activemq-client-5.15.0.jar:5.15.0] at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.15.0.jar:5.15.0] at java.lang.Thread.run(Thread.java:748)[:1.8.0_151]

3.

 java.lang.OutOfMemoryError：unable to create new native thread

 

At the same time, the number of threads in ActiveMQ service keeps skyrocketing, reaching 60000, resulting in memory overflow。I found that when the channel was inactive for too  long is kicked out by the broker, the old thread is not destroyed. At this time, new requests come in, and the thread will still be created continuously, resulting in memory overflow.

My requirement is that ActiveMQ broker can support 30000 client connections.How can I solve this problem? Can someone help me