RA does not reconnect correctly to a failed broker

• Key: AMQ-1779
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Rob Davies
• Reporter: Kai Hudalla
• Component/s: Connector
• Affects Version/s: 5.0.0, 5.1.0
• Fix Version/s: 5.2.0

Description

The Resource Adapter does not cleanly reconnect to a failed broker:

• it uses stale sessions to deliver messages to an Endpoint resulting in messages not being delivered at all anymore after the connection has been re-established
• the code for connecting/reconnecting to the broker is not synchronized properly leading to race conditions/deadlocks in some situations

Kai

Comments

[Kai Hudalla]

The work done by David Jencks on the 4.x branch in AMQ-1632 still has some issues regarding synchronization:

• When the RA tries to connect to the broker, it does so by subsequently scheduling the connectWork using the WorkManager. However, the monitor on connectWork may be lost when the connectWork is re-scheduled after an unsuccessful attempt, making it possible that the getConnection() method gets the monitor and returns null instead of the re-established connection.
• The code for registering a ConnectionConsumer on the ActiveMQConnection leads to a deadlock if there are some messages left to be delivered from before the broker has failed. In that case, the ServerSessionPoolImpl.createServerSessionImpl() method will be triggered by the next message delivery. However, the createServerSessionImpl() method needs to wait for the EndpointWorker.getConnection() method to return which is also synchronized on the connectWork that is trying to register the ConnectionConsumer. The problem here is that the constructor of the ActiveMQConnectionConsumer uses syncSendPacket() instead of asyncSendPacket() in order to register the consumer with the broker. Thus the instantiation of the ActiveMQConnectionConsumer blocks until the first message has been delivered, leading to the deadlock.

Kai

[Kai Hudalla]

I have changed the connectWork to loop until it finally gets a connection to the broker instead of re-scheduling itself after a failed connection attempt. This way, the monitor on connectWork is kept until the connection succeeds. In order to make this work, the ActiveMQConnectionConsumer needs to use asyncSendPacket() for registering itself with the consumer to avoid a deadlock on the connectWork.

The patch also includes David's fix for the prefetchSize calculation in the EndpointWorker.

Kai

[Rob Davies]

Patch applied by SVN revision 664079 and 664082