[AMQ-5599] Broker start thread hangs forever in async mode when startup of PersistenceAdapter is very fast

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.10.0
• Fix Version/s: 5.12.0
• Component/s: Broker
• Assignee: Unassigned
• Reporter: Ward V
• Created: 19/Feb/15 14:24
• Updated: 02/Jun/15 15:38
• Resolved: 02/Jun/15 15:38

Description

We encountered a situation where the startup of our broker fails sometimes. We investigated this and have seen that this is caused by a race condition.
When the BrokerService is configured to be async, the starting of the broker will hang when the starting of the PersistenceAdapter is faster than expected.

The ActiveMQ code will first start the persistence adapter, and then start the broker. When startup is configured to be 'async', these actions will happen async (in separate threads). Since these calls are async, we can not garuantee the order in which they will be finished. Although the ActiveMQ code now relies on the fact that the starting of the persistence adapter takes longer, and it waits in the starting of the broker for the starting of the persistence adapter to finish. When the starting of the persistence adapter already finished before the starting of the broker starts waiting for it, it will wait forever (and thus the broker cannot get started).

In the following snippet from org.apache.activemq.broker.BrokerService this becomes clear:

'startPersistenceAdapter' is fast and already notifies: persistenceAdapterLock.notifyAll();.
'startBroker' will wait for notification but no one will ever notify it again: persistenceAdapterLock.wait();

—

startPersistenceAdapter(startAsync);
startBroker(startAsync);

private void startPersistenceAdapter(boolean async) throws Exception {
if (async) {
new Thread("Persistence Adapter Starting Thread") {
@Override
public void run() {
try

{ doStartPersistenceAdapter(); }

catch (Throwable e)

{ startException = e; }

finally {
synchronized (persistenceAdapterLock)

{ persistenceAdapterLock.notifyAll(); }

}
}
}.start();
} else

{ doStartPersistenceAdapter(); }

}

private void startBroker(boolean async) throws Exception {
if (async) {
new Thread("Broker Starting Thread") {
@Override
public void run() {
try {
synchronized (persistenceAdapterLock)

{ persistenceAdapterLock.wait(); }

doStartBroker();
} catch (Throwable t)

{ startException = t; }

}
}.start();
} else

{ doStartBroker(); }

}