[AMQ-4328] Cross talk over duplex network connection can lead to blocking

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.7.0, 5.8.0
• Fix Version/s: 5.9.0
• Component/s: Connector
• Labels: bridge duplex hang network
• Assignee: Gary Tully
• Reporter: Gary Tully
• Created: 18/Feb/13 15:21
• Updated: 28/Mar/17 13:55
• Resolved: 18/Feb/13 15:34

Description

with active forwarding in both directions a duplex network connector can block. in 5.8, threads of the form:

"ActiveMQ BrokerService[xx] Task-10" daemon prio=10 tid=0xb35d1c00 nid=0xc64 runnable [0xb3369000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
	at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
	at java.io.DataOutputStream.flush(DataOutputStream.java:106)
	at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:176)
	at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:322)
	at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:304)
	at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
	at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
	at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
	at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
	at org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(DemandForwardingBridgeSupport.java:994)
	at org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:201)
	at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
	at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
	at org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)
	at org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:127)
	- locked <0x647f4650> (a java.util.concurrent.atomic.AtomicBoolean)
	at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:104)
	at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
	at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
	at org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1378)
	at org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:897)
	at org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:943)
	at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:129)
	at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:47)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

"ActiveMQ Transport: tcp:///xx:61616@40803" prio=10 tid=0xb3525400 nid=0xbec waiting on condition [0xb3276000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x64657028> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
	at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
	at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
	at org.apache.activemq.network.DemandForwardingBridgeSupport$7.onCompletion(DemandForwardingBridgeSupport.java:630)
	at org.apache.activemq.transport.FutureResponse.set(FutureResponse.java:65)
	at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:109)
	at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
	at org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)
	at org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:127)
	- locked <0x647f4c80> (a java.util.concurrent.atomic.AtomicBoolean)
	at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:104)
	at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
	at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
	at org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1378)
	at org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:897)
	at org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:857)
	at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:186)
	at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
	at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
	at org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)
	at org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:127)
	- locked <0x647e6528> (a java.util.concurrent.atomic.AtomicBoolean)
	at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:104)
	at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
	at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
	at org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:622)
	at org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:219)
	at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
	at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
	at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
	at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:288)
	at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
	at java.lang.Thread.run(Thread.java:662)

Issue Links

• breaks: AMQ-6640 network of brokers, duplex connector, network consumers not registered on destination - hung bridge
• is related to: AMQ-4276 Don't restart the network bridge on race condition where respondent sends to a temp dest that has just been deleted
• is related to: AMQ-5342 Deadlock in network of brokers simple architecture