Major Description
While recovering from a period of network instability, our application deadlocked in ActiveMQ code.
Found one Java-level deadlock:
=============================
"ActiveMQ Connection Executor: tcp:///<ip address>:61616@51026":
waiting to lock monitor 0x00007f60fc0060b8 (object 0x0000000773f00308, a java.lang.Object),
which is held by "privateJmsInPriority.container-1"
"privateJmsInPriority.container-1":
waiting to lock monitor 0x00007f610840dbc8 (object 0x0000000773f99930, a org.apache.activemq.ActiveMQConnection),
which is held by "ActiveMQ Connection Executor: tcp:///<ip address>:61616@51026"
Java stack information for the threads listed above:
===================================================
"ActiveMQ Connection Executor: tcp:///<ip address>:61616@51026":
at org.apache.activemq.SimplePriorityMessageDispatchChannel.close(SimplePriorityMessageDispatchChannel.java:154)
- waiting to lock <0x0000000773f00308> (a java.lang.Object)
at org.apache.activemq.ActiveMQMessageConsumer.dispose(ActiveMQMessageConsumer.java:800)
at org.apache.activemq.ActiveMQSession.dispose(ActiveMQSession.java:689)
- locked <0x0000000773f003d8> (a org.apache.activemq.ActiveMQSession)
at org.apache.activemq.ActiveMQConnection.close(ActiveMQConnection.java:659)
- locked <0x0000000773f99930> (a org.apache.activemq.ActiveMQConnection)
at org.springframework.jms.connection.SingleConnectionFactory.closeConnection(SingleConnectionFactory.java:456)
at org.springframework.jms.connection.SingleConnectionFactory.resetConnection(SingleConnectionFactory.java:345)
- locked <0x0000000773f9a590> (a java.lang.Object)
at org.springframework.jms.connection.CachingConnectionFactory.resetConnection(CachingConnectionFactory.java:207)
at org.springframework.jms.connection.SingleConnectionFactory.onException(SingleConnectionFactory.java:323)
at org.springframework.jms.connection.SingleConnectionFactory$AggregatedExceptionListener.onException(SingleConnectionFactory.java:673)
- locked <0x0000000773f9a590> (a java.lang.Object)
at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:1976)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"privateJmsInPriority.container-1":
at org.apache.activemq.ActiveMQConnection.getScheduler(ActiveMQConnection.java:2519)
- waiting to lock <0x0000000773f99930> (a org.apache.activemq.ActiveMQConnection)
at org.apache.activemq.ActiveMQSession.getScheduler(ActiveMQSession.java:2074)
at org.apache.activemq.ActiveMQMessageConsumer.rollback(ActiveMQMessageConsumer.java:1241)
- locked <0x0000000773f01030> (a java.util.LinkedList)
- locked <0x0000000773f00308> (a java.lang.Object)
at org.apache.activemq.ActiveMQMessageConsumer$5.afterRollback(ActiveMQMessageConsumer.java:1032)
at org.apache.activemq.TransactionContext.afterRollback(TransactionContext.java:157)
at org.apache.activemq.TransactionContext.commit(TransactionContext.java:332)
at org.apache.activemq.ActiveMQSession.commit(ActiveMQSession.java:561)
at sun.reflect.GeneratedMethodAccessor897.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.jms.connection.CachingConnectionFactory$CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:386)
at com.sun.proxy.$Proxy49.commit(Unknown Source)
at org.springframework.jms.support.JmsUtils.commitIfNecessary(JmsUtils.java:217)
at org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:761)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:665)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:319)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1166)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1158)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1055)
at java.lang.Thread.run(Thread.java:745)
Found 1 deadlock.