[AMQ-4150] in pure master/slave, when the network is not stable, neither master nor slave can provide service to client

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 5.7.0
• Fix Version/s: 5.8.0
• Component/s: Broker, Transport

Environment

CentOS release 5.6 (Final) java version "1.6.0_30"

• Assignee: Robert Davies
• Reporter: Wei Wei
• Created: 31/Oct/12 09:10
• Updated: 13/Nov/12 09:33
• Resolved: 13/Nov/12 09:33

Description

We use pure master/slave. When the network between master and slave is not stable, the connection between master and slave shutdown, then the slave try to startAllConnectors because of the follow exception:

2012-10-25 18:48:12,083 | ERROR | Network connection between vm://172_17_7_92_61619#0 and tcp:///172.17.7.91:61618 shutdown: Connection reset | org.apache.activemq.broker.ft.MasterConnector | ActiveMQ Transport: tcp:///172.17.7.91:61618
java.net.SocketException: Connection reset
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
	at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
	at java.io.DataOutputStream.flush(DataOutputStream.java:106)
	at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:184)
	at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:262)
	at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:244)
	at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
	at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
	at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
	at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
	at org.apache.activemq.broker.ft.MasterConnector.serviceRemoteCommand(MasterConnector.java:268)
	at org.apache.activemq.broker.ft.MasterConnector$2.onCommand(MasterConnector.java:138)
	at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
	at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
	at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
	at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:229)
	at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:222)
	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:204)
	at java.lang.Thread.run(Thread.java:662)

After the exception the follow log was printed:

2012-10-25 18:48:12,084 | WARN  | Master Failed - starting all connectors | org.apache.activemq.broker.BrokerService | ActiveMQ Transport: tcp:///172.17.7.91:61618

it seems that the connector started properly, but it can not provide service to client.

After a while, another exception occured:

2012-10-25 18:49:01,212 | ERROR | Network connection between vm://172_17_7_92_61619#0 and tcp:///172.17.7.91:61618 shutdown: Broken pipe | org.apache.activemq.broker.ft.MasterConnector | InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@351dd9b3
java.net.SocketException: Broken pipe
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
	at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
	at java.io.DataOutputStream.flush(DataOutputStream.java:106)
	at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:184)
	at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:262)
	at org.apache.activemq.transport.AbstractInactivityMonitor.access$500(AbstractInactivityMonitor.java:40)
	at org.apache.activemq.transport.AbstractInactivityMonitor$3.run(AbstractInactivityMonitor.java:151)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

this exception lead to the slave trying to startAllConnectors again, this time the follow exception occured:

2012-10-25 18:49:26,233 | ERROR | Failed to startAllConnectors | org.apache.activemq.broker.BrokerService | InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@351dd9b3
java.io.IOException: Transport Connector could not be registered in JMX: Failed to bind to server socket: nio://0.0.0.0:61619 due to: java.net.BindException: Address already in use
	at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:27)
	at org.apache.activemq.broker.BrokerService.registerConnectorMBean(BrokerService.java:1836)
	at org.apache.activemq.broker.BrokerService.startTransportConnector(BrokerService.java:2327)
	at org.apache.activemq.broker.BrokerService.startAllConnectors(BrokerService.java:2243)
	at org.apache.activemq.broker.BrokerService.masterFailed(BrokerService.java:446)
	at org.apache.activemq.broker.ft.MasterConnector.shutDown(MasterConnector.java:345)
	at org.apache.activemq.broker.ft.MasterConnector.serviceRemoteException(MasterConnector.java:250)
	at org.apache.activemq.broker.ft.MasterConnector$2.onException(MasterConnector.java:144)
	at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
	at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
	at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
	at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
	at org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160)
	at org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:268)
	at org.apache.activemq.transport.AbstractInactivityMonitor$3.run(AbstractInactivityMonitor.java:154)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Failed to bind to server socket: nio://0.0.0.0:61619 due to: java.net.BindException: Address already in use
	at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
	at org.apache.activemq.transport.tcp.TcpTransportServer.bind(TcpTransportServer.java:138)
	at org.apache.activemq.transport.tcp.TcpTransportFactory.doBind(TcpTransportFactory.java:60)
	at org.apache.activemq.transport.TransportFactory.bind(TransportFactory.java:131)
	at org.apache.activemq.broker.TransportConnector.createTransportServer(TransportConnector.java:329)
	at org.apache.activemq.broker.TransportConnector.getServer(TransportConnector.java:155)
	at org.apache.activemq.broker.TransportConnector.asManagedConnector(TransportConnector.java:110)
	at org.apache.activemq.broker.BrokerService.registerConnectorMBean(BrokerService.java:1831)
	... 16 more
Caused by: java.net.BindException: Address already in use
	at sun.nio.ch.Net.bind(Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
	at org.apache.activemq.transport.nio.NIOTransportFactory$2.createServerSocket(NIOTransportFactory.java:69)
	at org.apache.activemq.transport.tcp.TcpTransportServer.bind(TcpTransportServer.java:134)
	... 22 more

At last, neither the master nor the slave can provide service to client.

Issue Links

• is superceded by: AMQ-4165 Remove pure master/slave functionality