[AMQ-1956] NPE during broker shutdown when useDatabaseLock="false"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.1.0
• Fix Version/s: 5.2.0
• Component/s: Message Store
• Assignee: Gary Tully
• Reporter: Chris Pettitt
• Created: 26/Sep/08 17:37
• Updated: 08/Oct/08 08:49
• Resolved: 02/Oct/08 11:50

Description

Steps:

1. Create a broker with the persistence adapter set to not use database locking (useDatabaseLock="false")
2. Start broker
3. Stop broker (but keep the process running, as in a servlet container)

Result:

In the logs, I see the following error every 30 seconds:

2008/09/25 15:23:55.506 INFO [org.apache.activemq.store.jdbc.JDBCPersistenceAdapter] No longer able to keep the exclusive lock so giving up being a master
2008/09/25 15:23:55.506 WARN [org.apache.activemq.store.jdbc.JDBCPersistenceAdapter] Failed to stop broker
2008/09/25 15:24:25.504 ERROR [org.apache.activemq.store.jdbc.DefaultDatabaseLocker] Failed to update database lock: java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:102)
at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:458)
at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$3.run(JDBCPersistenceAdapter.java:260)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:613)

Analysis:

During startup, JDBCPersistenceAdapter only initializes the database locker if useDatabaseLock if true. This is done through lazy initialization by calling getDatabaseLocker (at ~ line 172):

if (isUseDatabaseLock()) {
DatabaseLocker service = getDatabaseLocker();
if (service == null)

{ LOG.warn("No databaseLocker configured for the JDBC Persistence Adapter"); }

else

{ service.start(); }

}

During shutdown, JDBCPersistenceAdapter calls getDatabaseLocker() to shut it down, but it does not check if isUseDatabaseLock() is true in this case:

public synchronized void stop() throws Exception {
if (clockTicket != null)

{ clockTicket.cancel(true); clockTicket = null; }

if (clockDaemon != null)

{ clockDaemon.shutdown(); clockDaemon = null; }

DatabaseLocker service = getDatabaseLocker();
if (service != null)

{ service.stop(); }

}

This actually causes database locker to be initialized and it subsequently lazy-initializes an executor to run a task which calls keepAlive(...). The executor threads are set as daemon threads which prevents this issue from showing up when the lifetime of the process is the same as the broker. When the broker is deployed in an app server which can outlive the broker then the above error is logged every 30 seconds.

I'm attaching one way to solve this problem without having to check everywhere if useDatabaseLock="false".

Attached:

BrokerStopFailure.java - sample program that reproduces this bug
NoLockerJDBCPersistenceAdapter.java - sample extension to JDBCPersistenceAdapter that fixes this bug