can get a shutdown hang when using embedded broker with VM transport along with the activemq-web module

• Key: AMQ-813
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Hiram Chirino
• Reporter: James Strachan
• Fix Version/s: 4.1.0, 4.0.2

Description

See attached thread dump. I think we just need to use a timeout on the close operations (such as to close consumers, sessions, producers)

Full thread dump Java HotSpot(TM) Client VM (1.5.0_04-b05 mixed mode, sharing):

"Shutdown" prio=1 tid=0x08385960 nid=0x5e99 in Object.wait() [0xaed7d000..0xaed7e130]
at java.lang.Object.wait(Native Method)

• waiting on <0x88af01c8> (a edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar)
  at java.lang.Object.wait(Object.java:474)
  at edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar.await(CondVar.java:75)
• locked <0x88af01c8> (a edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar)
  at edu.emory.mathcs.backport.java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:318)
  at org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:41)
  at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:72)
  at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1130)
  at org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1660)
  at org.apache.activemq.ActiveMQMessageConsumer.close(ActiveMQMessageConsumer.java:516)
  at org.apache.activemq.web.WebClient.closeConsumers(WebClient.java:135)
• locked <0x893e6558> (a org.apache.activemq.web.WebClient)
  at org.apache.activemq.web.WebClient.close(WebClient.java:145)
• locked <0x893e6558> (a org.apache.activemq.web.WebClient)
  at org.apache.activemq.web.WebClient.valueUnbound(WebClient.java:318)
  at org.mortbay.jetty.servlet.AbstractSessionManager$Session.unbindValue(AbstractSessionManager.java:899)
  at org.mortbay.jetty.servlet.AbstractSessionManager$Session.invalidate(AbstractSessionManager.java:755)
• locked <0x893caa88> (a org.mortbay.jetty.servlet.HashSessionManager$Session)
  at org.mortbay.jetty.servlet.AbstractSessionManager.doStop(AbstractSessionManager.java:551)
  at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:63)
  at org.mortbay.jetty.servlet.SessionHandler.doStop(SessionHandler.java:124)
  at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:63)
  at org.mortbay.jetty.handler.HandlerWrapper.doStop(HandlerWrapper.java:131)
  at org.mortbay.jetty.handler.ContextHandler.doStop(ContextHandler.java:467)
  at org.mortbay.jetty.webapp.WebAppContext.doStop(WebAppContext.java:477)
  at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:63)
  at org.mortbay.jetty.handler.HandlerCollection.doStop(HandlerCollection.java:173)
  at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:63)
  at org.mortbay.jetty.handler.HandlerCollection.doStop(HandlerCollection.java:173)
  at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:63)
  at org.mortbay.jetty.handler.HandlerWrapper.doStop(HandlerWrapper.java:131)
  at org.mortbay.jetty.Server.doStop(Server.java:242)
  at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:63)
  at org.mortbay.jetty.Server$ShutdownHookThread.run(Server.java:450)

Comments

[Hiram Chirino]

Fixed:
http://svn.apache.org/viewvc/incubator/activemq/trunk/activemq-core/src/main/java/org/apache/activemq/transport/vm/VMTransport.java?r1=415306&r2=420714&pathrev=420714

The VMTransport's oneway method was dropping the send request when the peer was disconnected like in the case where the broker is shutdown at the same time that the client is shutdown.

[Gray Watson]

This seems to generate another fault. With 4.1.0.0-fuse, I'm get a whole ton of the following, possibly in an infinite loop:

2006-10-05 16:07:52,178 [DefaultMessageListenerContainer-45] ERROR org.springframework.jms.listener.DefaultMessageListenerContainer - Setup of JMS message listener invoker failed - trying to recover
javax.jms.JMSException: Peer disconnected.
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:57)
at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1119)
at org.apache.activemq.ActiveMQSession.<init>(ActiveMQSession.java:225)
at org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:278)
at org.apache.activemq.pool.SessionPool.createSession(SessionPool.java:112)
at org.apache.activemq.pool.SessionPool.makeObject(SessionPool.java:80)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:840)
at org.apache.activemq.pool.SessionPool.borrowSession(SessionPool.java:55)
at org.apache.activemq.pool.ConnectionPool.createSession(ConnectionPool.java:70)
at org.apache.activemq.pool.PooledConnection.createSession(PooledConnection.java:129)
at org.springframework.jms.listener.AbstractMessageListenerContainer.createSession(AbstractMessageListenerContainer.java:1000)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.initResourcesIfNecessary(DefaultMessageListenerContainer.java:882)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:870)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:824)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException: Peer disconnected.
at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:81)
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:44)
at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:58)
at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1117)
... 14 more

[Hiram Chirino]

Hi Gary,

Please open a new issue since ti seems like a different problem. Also, a small web app test case would help us be able to reproduce the issue.