[AMQ-2774] Network of brokers : Multicast discovery stopped to work

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.2.0
• Fix Version/s: 5.4.0

Environment

Linux

• Assignee: Gary Tully
• Reporter: Eric
• Created: 11/Jun/10 15:30
• Updated: 10/Feb/11 10:59
• Resolved: 28/Jul/10 10:47

Description

Hi everybody

I experiment a big problem with the multicast discovery algorithm, in a network of brokers topology.
In some conditions, a broker can't reestablish a distant connection even if the distant broker is restarted.

I have the log traces that would help to identify the origin of the problem.

When there is no discovery/connection error, I can see these 2 lines in the activemq log file

#08 Jun 2010 14:31:30,639 INFO [Multicast Discovery Agent Notifier] org.apache.activemq.network.DiscoveryNetworkConnector
Establishing network connection between from vm://ACCLU-tpnocp04v to tcp://tpnocp09v-bus:13100?useLocalHost=false

#08 Jun 2010 14:31:30,692 INFO StartLocalBridge: localBroker=vm://ACCLU-tpnocp04v#26 org.apache.activemq.network.DemandForwardingBridge
Network connection between vm://ACCLU-tpnocp04v#26 and tcp://tpnocp09v-bus/10.18.126.28:13100(MOM-tpnocp09v) has been established.

When the connection is broken, I can see this line in the log.

#11 Jun 2010 12:37:32,585 INFO [Multicast Discovery Agent Notifier] org.apache.activemq.network.DemandForwardingBridge
ACCLU-tpnocp04v bridge to MOM-tpnocp09v stopped

Then the current ACCLU-tpnocp04v broker tries to reestablish the connection :

#11 Jun 2010 12:37:34,475 INFO [Multicast Discovery Agent Notifier] org.apache.activemq.network.DiscoveryNetworkConnector
Establishing network connection between from vm://ACCLU-tpnocp04v to tcp://tpnocp09v-bus:13100?useLocalHost=false

But, here, the second line of the log ("has been established") doesn't appear in the log file !! I don't know exactly if the connection is up or not.

Then the connection is broken again (look at "Unknown" instead of "MOM-tpnocp09v".

#11 Jun 2010 13:33:58,655 WARN [ActiveMQ Transport: tcp://tpnocp09v-bus/10.18.126.28:13100] org.apache.activemq.network.DemandForwardingBridge
Network connection between vm://ACCLU-tpnocp04v#58 and tcp://tpnocp09v-bus/10.18.126.28:13100 shutdown due to a remote error: java.net.SocketException: Connection reset
#11 Jun 2010 13:33:58,657 INFO [NetworkBridge] org.apache.activemq.network.DemandForwardingBridge^M
ACCLU-tpnocp04v bridge to Unknown stopped

And, now, even if I restart the distant broker ( MOM-tpnocp09v ), no line (Establishing/Has been established) appears, and no network connection is reestablished between ACCLU-tpnocp04v and MOM-tpnocp09v. it seems that this ACCLU-tpnocp04v broker can no longer establish a connection with the MOM-tpnocp09v broker !!!

The production teams tell me that this problem seems not to be resolved in fuse-5.3.0.6 version.

Eric-AWL

Issue Links

• breaks: AMQ-3176 Potential deadlock in duplex network connector recreation, resulting in dangling connections
• incorporates: AMQ-3129 Can only have one duplex networkConnection per transportConnection