[AMQ-8416] Deadlock during failover transport reconnect

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 5.16.0
• Component/s: Transport

Environment

Amazon Linux 2, Apache Tomcat, Java 8

• Assignee: Jean-Baptiste Onofré
• Reporter: Geoff Denning
• Created: 24/Nov/21 21:31
• Updated: 07/Jul/23 11:05

Description

Using the ActiveMQ Failover Transport, we're seeing issues where our application randomly stops receiving messages from ActiveMQ until the application is restarted.

We ran a thread dump, and saw that a Java-level deadlock was detected between two ActiveMQ threads:

Found one Java-level deadlock:
=============================
"ActiveMQ Task-1":
  waiting to lock monitor 0x00007f8286d91a18 (object 0x00000000c06966d8, a java.lang.Object),
  which is held by "ActiveMQ Task-3"
"ActiveMQ Task-3":
  waiting to lock monitor 0x00007f8286d91ac8 (object 0x00000000c0afb960, a java.lang.Object),
  which is held by "ActiveMQ Task-1"

Java stack information for the threads listed above:
===================================================
"ActiveMQ Task-1":
    at org.apache.activemq.transport.failover.FailoverTransport.handleTransportFailure(FailoverTransport.java:274)
    - waiting to lock <0x00000000c06966d8> (a java.lang.Object)
    at org.apache.activemq.transport.failover.FailoverTransport$3.onException(FailoverTransport.java:224)
    at org.apache.activemq.transport.failover.FailoverTransport.propagateFailureToExceptionListener(FailoverTransport.java:1147)
    at org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1096)
    - locked <0x00000000c0afb960> (a java.lang.Object)
    at org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:149)
    - locked <0x00000000c0afb998> (a java.lang.Object)
    at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
    at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
"ActiveMQ Task-3":
    at java.lang.Object.wait(Native Method)
    at org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:623)
    - locked <0x00000000c0afb960> (a java.lang.Object)
    at org.apache.activemq.transport.failover.FailoverTransport.restoreTransport(FailoverTransport.java:853)
    at org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1022)
    - locked <0x00000000c06966d8> (a java.lang.Object)
    at org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:149)
    - locked <0x00000000c06966e8> (a java.lang.Object)
    at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
    at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Found 1 deadlock.