[AMQ-9482] Broker crashes after runaway threads spawn

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 5.17.6, 6.0.1
• Component/s: Broker

Environment

Bitnami created AMI in AWS

• Assignee: Unassigned
• Reporter: Tom Tichy
• Created: 19/Apr/24 11:28
• Updated: 27/May/24 16:47

Description

Running on Bitnami created AMI in AWS. The broker has about 7000 devices connected via MQTT. Each devices has its own topic name.

Broker stays up for about 4-5 days before being hobbled and unable to create any new tasks/accept any new connections.

(There is identical setup for staging environment with about 100 devices connected. It runs without any issues.)

I have troubleshot the cause to be the systemd task limit. The current `TasksMax` is 18100. When running normally, the number of tasks is about 300. Then (every 4-5 days) there is a quick spike to the max 18100 tasks and it stays there never coming back down. The result is that the broker just sits there, does nothing useful and keeps logging the following message

 

[659914.788s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, g
uardsize: 0k, detached.
[659914.788s][warning][os,thread] Failed to start the native thread for java.lang.Thread "ActiveMQ BrokerService[localhost] Task-281805"
ERROR | Scheduled task error
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.lang.Thread.start0(Native Method) ~[?:?]
        at java.lang.Thread.start(Thread.java:809) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364) ~[?:?]
        at org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:173) ~[activemq-client-6.0.1.jar:6.0.1]
        at org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:165) ~[activemq-client-6.0.1.jar:6.0.1]
        at org.apache.activemq.broker.region.Topic$7.run(Topic.java:820) ~[activemq-broker-6.0.1.jar:6.0.1]
        at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:39) ~[activemq-client-6.0.1.jar:6.0.1]
        at java.util.TimerThread.mainLoop(Timer.java:566) ~[?:?]
        at java.util.TimerThread.run(Timer.java:516) ~[?:?]
Exception in thread "ActiveMQ Broker[localhost] Scheduler" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.base/java.lang.Thread.start0(Native Method)
        at java.base/java.lang.Thread.start(Thread.java:809)
        at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945)
        at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
        at org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:173)
        at org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:165)
        at org.apache.activemq.broker.region.Topic$7.run(Topic.java:820)
        at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:39)
        at java.base/java.util.TimerThread.mainLoop(Timer.java:566)
        at java.base/java.util.TimerThread.run(Timer.java:516)
 

 

The start command is 

/opt/bitnami/java/bin/java -Xms2G -Xmx4G -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/opt/bitnami/activemq/conf/login.config -Dorg.apache.activemq.UseDedicatedTaskRunner=false -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Djava.io.tmpdir=/opt/bitnami/activemq/tmp --add-reads=java.xml=java.logging --add-opens java.base/java.security=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.naming/javax.naming.spi=ALL-UNNAMED --add-opens java.rmi/sun.rmi.transport.tcp=ALL-UNNAMED --add-opens java.base/java.util.concurrent=ALL-UNNAMED --add-opens java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.http=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.https=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.jar=ALL-UNNAMED --add-exports=jdk.xml.dom/org.w3c.dom.html=ALL-UNNAMED --add-exports=jdk.naming.rmi/com.sun.jndi.url.rmi=ALL-UNNAMED -Dactivemq.classpath=/opt/bitnami/activemq/conf:/opt/bitnami/activemq/../lib/: -Dactivemq.home=/opt/bitnami/activemq -Dactivemq.base=/opt/bitnami/activemq -Dactivemq.conf=/opt/bitnami/activemq/conf -Dactivemq.data=/opt/bitnami/activemq/data -Djolokia.conf=file:/opt/bitnami/activemq/conf/jolokia-access.xml -jar /opt/bitnami/activemq/bin/activemq.jar start 

During the error condition, I am able to collect broker information via jolokia: brokerInfo-after-crash-redacted.json