[AMQ-3993] NetworkBridge sometimes stops trying to reconnect after connection is lost

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.6.0
• Fix Version/s: 5.8.0

Environment

using static:// networkConnector (i.e. SimpleDiscoveryAgent)

• Assignee: Timothy A. Bish
• Reporter: Ron Koerner
• Created: 27/Aug/12 10:52
• Updated: 02/May/13 02:30
• Resolved: 05/Nov/12 22:55

Description

After losing connection due to shutdown of the peer the broker tries to rebuild the connection once, fails again and stops trying afterwards.

While this also happens with a standard setup, it seems to happen much more often with a certain type of firewall which always accepts a connection, but closes it if the real destination cannot be reached.

This can be simulated by using a "socat" forwarder between the two brokers.

The problems seems to lie in the following sequence of events, a race condition and the use of event.failed in SimpleDiscoveryAgent.serviceFailed and bridges in DiscoveryNetworkConnector:

1. connection "failure" due to ShutdownInfo
   • event.failed=true
   • bridge is unregistered
2. start establishing a new connection
   • event.failed=false
   • bridge is not yet registered
3. second connection failure of the old connection due to EOF
   • not blocked, since event.failed==false
   • event.failed=true
   • bridge would be unregistered, but currently there is none
   • wait one second (continued below)
4. new connection is started
   • bridge is registered
5. receive multiple connection failures of the new connection
   • all blocked, since event.failed=true
6. continue after one second, try to establish a new connection
   • blocked, since bridge is already registered

To fix this problem a NetworkBridge should probably not be allowed to call SimpleDiscoveryAgent.serviceFailed more than once, since event.failed cannot keep track of multiple connections at one time.

The chain of events holds a lot of race conditions. If the second failure of the old connection occurs before the new connection is started (which seems to be the case most of the time) or the new connection's bridge is registered before the EOF occurs, the problem does not manifest.

Attached is a log excerpt with my comments about the state of event.failed and bridges.

Issue Links

• relates to: AMQ-4159 Race condition in SimpleDiscoveryAgent creates multiple concurrent threads attempting to connect to the same bridge --- can result in deadlock
• relates to: AMQ-4160 DiscoveryNetworkConnector can lose track of active bridges, resulting in permanent bridge failure or continued attempts to re-connect existing bridges