Deadlock in ActiveMQ close processing

• Key: GERONIMO-1493
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: David Jencks
• Reporter: Kevan Miller
• Component/s: ActiveMQ
• Affects Version/s: 1.0
• Fix Version/s: 1.x

Description

Poor use of synchronization in ActiveMQAsfEndpointWorker can prevent Geronimo from shutting down. The problem showed up in Jira 1422 (http://issues.apache.org/jira/browse/GERONIMO-1422). And is contained in the attached file (http://issues.apache.org/jira/secure/attachment/12321750/geronimo_shutdown_stdout.txt).

The following thread is attempting to reconnect to the broker:

"Thread-91" prio=7 tid=0x08358d50 nid=0x91 waiting on condition [c082f000..c082fd98]
at java.lang.Thread.sleep(Native Method)
at org.activemq.ra.ActiveMQAsfEndpointWorker.reconnect(ActiveMQAsfEndpointWorker.java:177)

• locked <0xc7c80470> (a org.activemq.ra.ActiveMQAsfEndpointWorker)
  at org.activemq.ra.ActiveMQAsfEndpointWorker.access$200(ActiveMQAsfEndpointWorker.java:40)
  at org.activemq.ra.ActiveMQAsfEndpointWorker$1.run(ActiveMQAsfEndpointWorker.java:105)
• locked <0xc7c7d138> (a org.activemq.ra.ActiveMQAsfEndpointWorker$1)
  at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:291)
  at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
  at java.lang.Thread.run(Thread.java:534)

While the following thread is attempting to close the EndpointWorker:

"Geronimo shutdown thread" prio=5 tid=0x08ed3030 nid=0x7f waiting for monitor entry [c07ed000..c07edd98]
at org.activemq.ra.ActiveMQAsfEndpointWorker.stop(ActiveMQAsfEndpointWorker.java:135)

• waiting to lock <0xc7c80470> (a org.activemq.ra.ActiveMQAsfEndpointWorker)
  at org.activemq.ra.ActiveMQResourceAdapter.endpointDeactivation(ActiveMQResourceAdapter.java:261)
  at org.apache.geronimo.connector.ResourceAdapterWrapper.endpointDeactivation(ResourceAdapterWrapper.java:92)
  at org.apache.geronimo.connector.ResourceAdapterWrapper$$FastClassByCGLIB$$4ab28e73.invoke(<generated>)
  at net.sf.cglib.reflect.FastMethod.invoke(FastMethod.java:53)
  at org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke(FastMethodInvoker.java:38)
  at org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:118)
  at org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:800)
  at org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java:57)
  at org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:36)
  at org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java:96)
  at org.apache.geronimo.connector.ResourceAdapterWrapper$$EnhancerByCGLIB$$168e117d.endpointDeactivation(<generated>)
  at org.apache.geronimo.connector.ActivationSpecWrapper.deactivate(ActivationSpecWrapper.java:109)
  at org.apache.geronimo.connector.ActivationSpecWrapper$$FastClassByCGLIB$$aaa078c1.invoke(<generated>)
  at net.sf.cglib.reflect.FastMethod.invoke(FastMethod.java:53)
  at org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke(FastMethodInvoker.java:38)
  at org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:118)
  at org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:800)
  at org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java:57)
  at org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:36)
  at org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java:96)
  at org.apache.geronimo.connector.ActivationSpecWrapper$$EnhancerByCGLIB$$17d592bb.deactivate(<generated>)
  at org.openejb.mdb.MDBContainer.doStop(MDBContainer.java:223)
  at org.apache.geronimo.gbean.runtime.GBeanInstance.destroyInstance(GBeanInstance.java:1079)
  at org.apache.geronimo.gbean.runtime.GBeanInstanceState.attemptFullStop(GBeanInstanceState.java:395)
  at org.apache.geronimo.gbean.runtime.GBeanInstanceState.stop(GBeanInstanceState.java:200)
  at org.apache.geronimo.gbean.runtime.GBeanInstance.stop(GBeanInstance.java:545)
  at org.apache.geronimo.kernel.basic.BasicKernel.stopGBean(BasicKernel.java:213)
  at org.apache.geronimo.gbean.runtime.GBeanInstanceState.stop(GBeanInstanceState.java:192)
  at org.apache.geronimo.gbean.runtime.GBeanInstance.stop(GBeanInstance.java:545)
  at org.apache.geronimo.kernel.basic.BasicKernel.stopGBean(BasicKernel.java:213)
  at org.apache.geronimo.gbean.runtime.GBeanInstanceState.stop(GBeanInstanceState.java:192)
  at org.apache.geronimo.gbean.runtime.GBeanInstance.stop(GBeanInstance.java:545)
  at org.apache.geronimo.kernel.basic.BasicKernel.stopGBean(BasicKernel.java:213)
  at org.apache.geronimo.kernel.config.ConfigurationManagerImpl$ShutdownHook.run(ConfigurationManagerImpl.java:287)
  at org.apache.geronimo.kernel.basic.BasicKernel.notifyShutdownHooks(BasicKernel.java:406)
  at org.apache.geronimo.kernel.basic.BasicKernel.shutdown(BasicKernel.java:383)
  at org.apache.geronimo.system.main.Daemon$1.run(Daemon.java:272)

ActiveMQAsfEndpointWorker.reconnect is synchronized and is using Thread.sleep within it's retry loop (not a good idea). Since the reconnect will never succeed (the broker has been stopped), this prevents Geronimo from ever closing. I've altered the shutdown ordering. So, it's less likely that the problem will occur, but the bug still exists...

IMO, the appropriate fix is to use Thread.wait() within reconnect(); maintain/monitor appropriate state (e.g. closed = true) within reconnect() and close(), and Thread.notify() in close().

Comments

[Hiram Chirino]

See
http://jira.activemq.org/jira/browse/AMQ-651

[David Jencks]

Correct amq x-ref is https://issues.apache.org/activemq/browse/AMQ-651 which is marked fixed.