[AMQ-3873] Occasional deadlock during startup

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.6.0
• Fix Version/s: 5.7.0

Environment

Suse Linux, CentOS Linux
Out-of-the-Box standalone broker with additional beans

• Assignee: Timothy A. Bish
• Reporter: Ron Koerner
• Created: 01/Jun/12 17:06
• Updated: 15/Jun/12 15:35
• Resolved: 15/Jun/12 15:35

Description

During startup I occasionally get deadlocks. I never had those on earlier versions including a 5.6-SNAPSHOT from January.

My activemq.xml contains a number of beans which each autowire the BrokerService and make connections to brokerService.getVmConnectorURI().toString()+"?async=false"

To avoid beans to be started faster than the broker and create a second broker by trying to connect, the broker is instantiated with start=false and a special bean listening for the Spring ContextRefreshedEvent will start the broker and run each beans connection methods.

Therefore a number of VmConnections will be done serially but in a rapid succession.

This fails at different points, but always with the following thread lock analysis:

Found one Java-level deadlock:
=============================
"ActiveMQ Task-3":
  waiting for ownable synchronizer 0x9f288120, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "BrokerService[smcufs02] Task-1"
"BrokerService[smcufs02] Task-1":
  waiting to lock monitor 0x0807a650 (object 0x9f2880d8, a java.util.concurrent.atomic.AtomicBoolean),
  which is held by "ActiveMQ Task-3"

Java stack information for the threads listed above:
===================================================
"ActiveMQ Task-3":
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x9f288120> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
        at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
        at org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1307)
        at org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:837)
        at org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:798)
        at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:152)
        at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
        at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
        at org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:135)
        at org.apache.activemq.transport.vm.VMTransport.start(VMTransport.java:156)
        - locked <0x9f2880d8> (a java.util.concurrent.atomic.AtomicBoolean)
        at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
        at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
        at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:914)
        - locked <0x9f2e4f98> (a org.apache.activemq.broker.TransportConnection)
        at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:227)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
"BrokerService[smcufs02] Task-1":
        at org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:114)
        - waiting to lock <0x9f2880d8> (a java.util.concurrent.atomic.AtomicBoolean)
        at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:103)
        at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
        at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
        at org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1307)
        at org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:837)
        at org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:872)
        at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
        at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

Found 1 deadlock.