[QPID-2871] QPID management is stuck in a waiting state for broker sync during rapid federation/defederation using the Java management API

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Later
• Affects Version/s: 0.6
• Fix Version/s: JIRA Cleanup
• Component/s: Java Common

Environment

RHEL 5.5, MRG-M 1,1, QPID Java Management API 0.6, 100GB hard drive, 4GB RAM, virtualized using VMWare

• Assignee: Unassigned
• Reporter: Jason Wong
• Created: 19/Sep/10 01:31
• Updated: 05/Sep/11 21:46
• Resolved: 05/Sep/11 21:46

Description

We used the QPID Java Management API to create a Java library to replicate the qpid-route CLI tool. When we rapidly federated, un-federated, and re-federated brokers, the broker's management component would become unresponsive to any other commands. Looking for locked or waiting threads, we noticed that the following class was consistently in a waiting state.

org.apache.qpid.console.Broker in the qpid-management-console.jar

Looking at the latest source we noticed that in the Broker.java waitForSync method the timeout check that throws a ConsoleException (line 498) is outside of the while loop (line 487-496) that is waiting for the broker to sync. We downloaded the source, put the timeout check inside the while loop, and compiled a new version of the qpid-management-console.jar. When we ran our original scenario, the broker's management component never became unresponsive. We did get some ConsoleExceptions thrown as we expected when we rapdily sent federation commands. The line numbers above were taken from revision 998162 (http://svn.apache.org/viewvc/qpid/trunk/qpid/java/management/console/src/main/java/org/apache/qpid/console/Broker.java?view=markup).

Updated code for the waitForSync method:
public void waitForSync(int timeout)
{
synchronized (lockObject)
{
long start = System.currentTimeMillis();
while (syncInFlight)
{
try

{ lockObject.wait(SYNC_TIME); }

catch (Exception e)

{ throw new ConsoleException(e); }

long duration = System.currentTimeMillis() - start;
if (duration > timeout)

{ throw new ConsoleException("Timeout waiting for Broker to Sync"); }

}
}
}