Broker is not re-connecting to a network of brokers after going down and then being brought back up

• Key: AMQ-639
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Critical
• Assignee: Hiram Chirino
• Reporter: Brian Diesenhaus
• Component/s: Broker
• Affects Version/s: 4.0 M4
• Fix Version/s: 4.0

Description

I have set up a network of brokers with the following configuration:

<beans xmlns="http://activemq.org/config/1.0">

<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>

<broker brokerName="bfe-${HOSTNAME}" useJmx="true" persistent="false" managementContext="#mc">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" memoryLimit="5000000"/>
</policyEntries>
</policyMap>
</destinationPolicy>
<memoryManager>
<usageManager limit="100000000"/>
</memoryManager>

<transportConnectors>
<transportConnector uri="tcp://localhost:61616" discoveryUri="multicast://bfe2"/>
</transportConnectors>

<networkConnectors>
<!-- by default just auto discover the other brokers -->
<networkConnector uri="multicast://bfe2"/>
</networkConnectors>
</broker>

<bean id="mc" class="org.apache.activemq.broker.jmx.ManagementContext">
<property name="connectorPort" value="1099"/>
<property name="connectorPath" value="/jmxrmibfe-1"/>
</bean>

</beans>

I then run a series of tests (producing and consuming on the network of brokers). Then I shut one broker down and then start it up again it can't see the other brokers in the network but the other brokers can see it.

Comments

[Hiram Chirino]

Thanks for the report! Fixed.

[Brian Diesenhaus]

I did some testing this morning and here is what I found:
1. The broker is re-connecting. Looking at the connection in the JMX console all the brokers can see each other. This was not the case before.
2. The brokers that were not brought down are throwing the below exception when the broker re-connects.
3. Running the test again after the broker comes up only about 1/3 of the message make it to the broker that was brought back up and then it .

INFO: Async error occurred: javax.jms.InvalidClientIDException: Broker: bfe-grieg - Client: NC_bfe-holst_inboundbfe-grieg already connected
javax.jms.InvalidClientIDException: Broker: bfe-grieg - Client: NC_bfe-holst_inboundbfe-grieg already connected
at org.apache.activemq.broker.region.RegionBroker.addConnection(RegionBroker.java:151)
at org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:64)
at org.apache.activemq.advisory.AdvisoryBroker.addConnection(AdvisoryBroker.java:67)
at org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:64)
at org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:76)
at org.apache.activemq.broker.AbstractConnection.processAddConnection(AbstractConnection.java:500)
at org.apache.activemq.command.ConnectionInfo.visit(ConnectionInfo.java:106)
at org.apache.activemq.broker.AbstractConnection.service(AbstractConnection.java:196)
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:62)
at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:88)
at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:70)
at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:75)
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:44)
at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:55)
at org.apache.activemq.network.DemandForwardingBridgeSupport.startLocalBridge(DemandForwardingBridgeSupport.java:175)
at org.apache.activemq.network.DemandForwardingBridgeSupport$3.run(DemandForwardingBridgeSupport.java:147)

BTW I wonder if the exception you are getting is related to this fix...

http://jira.activemq.org/jira/browse/AMQ-600

which was causing InvalidClientIDException to be thrown when network errors occur by handling the IOException in the wrong way

[James Strachan]

(BTW that issue was fixed in the last hour or so in SVN)

[Brian Diesenhaus]

I pulled the latest code with the fix for AMQ-600 and I am still seeing the following issues:
1. The brokers that were not brought down are still throwing the InvalidClientIDException when the broker that was brought down tries to re-connect.
2. Only about 1/3 of the message make it to the broker that was brought back up after running the test again.

[Hiram Chirino]

I've tried to setup a similar scenario. And I do see some message loss but when the broker is taken down but that can be expected since the messages are not persistent. Once the broker is brought back online, processing continues without message loss. When a broker is shutdown, any transient messages he had in it's queues will be dropped. Do you think this could explain the lost messages you've been seeing?

[Hiram Chirino]

Fixed.