[ARTEMIS-217] Deadlock between Topology updates and connection creation

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.1.0
• Assignee: Clebert Suconic
• Reporter: Clebert Suconic
• Created: 03/Sep/15 16:26
• Updated: 09/Sep/15 13:24
• Resolved: 09/Sep/15 13:24

Description

This was a rare case, it only happened in one specific environment:

"Thread-5 (ActiveMQ-client-netty-threads-1937173143)" #315 daemon prio=5 os_prio=0 tid=0x00007f5b5c020000 nid=0x361d waiting for monitor entry [0x00007f5bb9587000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.notifyNodeUp(ServerLocatorImpl.java:1413)

• waiting to lock <0x000000008832bbf8> (a org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl)
  at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$SessionFactoryTopologyHandler.notifyNodeUp(ClientSessionFactoryImpl.java:1275)
  at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.notifyTopologyChange(ActiveMQClientProtocolManager.java:491)
  at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handlePacket(ActiveMQClientProtocolManager.java:445)
  at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:559)
  at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:349)
• locked <0x0000000087145738> (a java.lang.Object)
  at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:331)
  at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1097)
  at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:68)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
  at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
  at java.lang.Thread.run(Thread.java:745)

versus:

"Thread-1 (ActiveMQ-server-ActiveMQServerImpl::Server 4-1766145591)" #466 prio=5 os_prio=0 tid=0x00007f5bf4f54000 nid=0x367d in Object.wait() [0x00007f5b5249e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

• waiting on <0x00000000e81ca680> (a io.netty.channel.DefaultChannelPromise)
  at java.lang.Object.wait(Object.java:502)
  at io.netty.util.concurrent.DefaultPromise.awaitUninterruptibly(DefaultPromise.java:286)
• locked <0x00000000e81ca680> (a io.netty.channel.DefaultChannelPromise)
  at io.netty.channel.DefaultChannelPromise.awaitUninterruptibly(DefaultChannelPromise.java:135)
  at io.netty.channel.DefaultChannelPromise.awaitUninterruptibly(DefaultChannelPromise.java:28)
  at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:604)
  at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.openTransportConnection(ClientSessionFactoryImpl.java:982)
  at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1026)
  at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.establishNewConnection(ClientSessionFactoryImpl.java:1205)
  at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:843)
• locked <0x00000000e685c1b0> (a java.lang.Object)
  at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:745)
  at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:231)
  at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:791)
• locked <0x000000008832bbf8> (a org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl)
  at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:649)
  at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:636)
  at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl$4.run(ServerLocatorImpl.java:1479)
  at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:94)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)